-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: rewrite marvin docs to use spack envs
- Loading branch information
James
committed
May 25, 2023
1 parent
86abebe
commit a7dec3a
Showing
1 changed file
with
115 additions
and
55 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,83 +1,143 @@ | ||
|
|
||
| Installing on Marvin | ||
| ***************************************** | ||
| ******************** | ||
|
|
||
| This is a temporary solution to installing PSC on Marvin. A better solution would use a spack environment. | ||
| There are two main approaches using Spack. The first option is using Spack to directly install psc (i.e., ``spack install psc``) after a few preparatory steps, and then later manually loading PSC's immediate dependencies to actually compile (e.g., ``spack load cmake``). The second approach is outlined below and results in a Spack environment in which PSC can be compiled. | ||
|
|
||
| Starting from scratch | ||
| ===================== | ||
|
|
||
| First we need git and spack: | ||
|
|
||
| .. code-block:: sh | ||
| $ module load git-gcc8/2.24.0 | ||
| $ git clone https://github.com/spack/spack.git | ||
| Loading Git | ||
| =========== | ||
|
|
||
| To enable spack commands, run | ||
| .. code-block:: bash | ||
| $ module load git-gcc8 | ||
| .. code-block:: sh | ||
| Installing Spack | ||
| ================ | ||
|
|
||
| $ . ${HOME}/spack/share/spack/setup-env.sh | ||
| In your home directory: | ||
|
|
||
| or just add the line to ``.bash_profile``. | ||
| .. code-block:: bash | ||
| $ git clone https://github.com/spack/spack.git | ||
| To clone PSC, one may need to set up an RSA key (via ``ssh-keygen``; hit "enter" on all prompts). Copy the full contents of ``id_rsa.pub`` (ie, the output of ``cat`` below) to a new RSA key on GitHub. | ||
| Note: the version of Spack used here was commit 2d77e44f6f57dcd1529c1ce712acfc6a60f47206, "Pcluster local buildcache". Release 1.4.1 should also be fine. You can swich spack versions by checkout out the desired commit in the spack repo. | ||
|
|
||
| Enable ``spack`` commands by running: | ||
|
|
||
| .. code-block:: bash | ||
| $ . ~/spack/share/spack/setup-env.sh | ||
| It is suggested that you add the above line to your ``.bash_profile``. | ||
|
|
||
| Fixing Spack | ||
| ============ | ||
|
|
||
| Spack fails to do several things out-of-the-box. First, fix ``~/.spack/cray/compilers.yaml`` by specifying the paths for your compiler. The default is gcc8.2.0, as below. Either delete all other compilers in this yaml, specify the appropriate paths for all of the compilers, or specify ``%[email protected]`` in your Spack specs. | ||
|
|
||
| .. code-block:: yaml | ||
| # file: ~/.spack/cray/compilers.yaml | ||
| - compiler: | ||
| spec: gcc@=8.2.0 | ||
| paths: | ||
| cc: /cm/shared/apps/gcc8/8.2.0/bin/gcc | ||
| cxx: /cm/shared/apps/gcc8/8.2.0/bin/g++ | ||
| f77: /cm/shared/apps/gcc8/8.2.0/bin/gfortran | ||
| fc: /cm/shared/apps/gcc8/8.2.0/bin/gfortran | ||
| flags: {} | ||
| operating_system: rhel7 | ||
| target: any | ||
| modules: | ||
| - PrgEnv-gnu | ||
| - gcc/8.2.0 | ||
| environment: {} | ||
| extra_rpaths: [] | ||
| Second, it is important to use Marvin's built-in slurm. Edit ``~/.spack/packages.yaml`` like so: | ||
|
|
||
| .. code-block:: yaml | ||
| # file: ~/.spack/packages.yaml | ||
| packages: | ||
| slurm: | ||
| externals: | ||
| - spec: [email protected] | ||
| prefix: /cm/shared/apps/slurm/18.08.9 | ||
| Note the prefix (and version) should agree with the results of ``module show slurm``. Specifically, the prefix needs to be a path to a folder containing both ``include/`` and ``lib64/`` (or symlinks thereto) for slurm. | ||
|
|
||
| Setting Up Spack Environment | ||
| ============================ | ||
|
|
||
| Make a Spack environment named "psc" (or whatever) and edit the config to match the yaml below. As mentioned above, you may want to add ``%[email protected]`` (or whatever compiler version you're using) to each of these specs. | ||
|
|
||
| .. code-block:: bash | ||
| $ spack env create psc | ||
| $ spack -e psc config edit | ||
| .. code-block:: yaml | ||
| # file: ~/spack/var/spack/environments/psc/spack.yaml | ||
| spack: | ||
| specs: | ||
| - "[email protected]:" | ||
| - gtensor | ||
| - hdf5+hl | ||
| - "[email protected]:2.8.3" | ||
| - "[email protected]:" | ||
| - "openmpi+pmi schedulers=slurm" | ||
| view: true | ||
| concretizer: | ||
| unify: true | ||
| Then concretize and install the packages (this takes a while). There should be a single warning after the install about skipping slurm, an external package. | ||
|
|
||
| .. code-block:: bash | ||
| $ spack -e psc concretize | ||
| $ spack -e psc install | ||
| Cloning PSC | ||
| =========== | ||
|
|
||
| .. code-block:: sh | ||
| You may need to set up an RSA key (via ``ssh-keygen``; hit "enter" on all prompts). Copy the full contents of ``id_rsa.pub`` (ie, the output of ``cat`` below) to a new RSA key on GitHub. | ||
|
|
||
| .. code-block:: bash | ||
| $ ssh-keygen | ||
| $ cat .ssh/id_rsa.pub | ||
| $ cat ~/.ssh/id_rsa.pub | ||
| It should now be possible to do | ||
|
|
||
| .. code-block:: sh | ||
| Clone PSC wherever you want it: | ||
|
|
||
| .. code-block:: bash | ||
| $ git clone [email protected]:psc-code/psc.git | ||
| $ git clone [email protected]:psc-code/psc-infrastructure.git | ||
| Continuing with spack... | ||
| .. code-block:: sh | ||
| Compiling PSC | ||
| ============= | ||
|
|
||
| $ spack repo add ${HOME}/psc-infrastructure/spack/psc | ||
| Before compiling the code, you must first activate the spack environment: | ||
|
|
||
| At this point, copy the files in ``psc-infrastructure/TODO`` to spack (TODO: currently, one needs to copy the contents of .spack from another user): | ||
| .. code-block:: bash | ||
| $ spack env activate psc | ||
| .. code-block:: sh | ||
| This command is slow and may interfere with subsequent slurm commands, so it is *not* recommended that you put this in your bash profile. Just activate it when working on the code. | ||
|
|
||
| $ cp -a psc-infrastructure/TODO/* .spack/ | ||
| To actually compile, go into the cloned repository and run | ||
|
|
||
| Finally, to complete installation of PSC, run | ||
|
|
||
| .. code-block:: sh | ||
| $ spack install psc | ||
| (this will take a while). | ||
| .. code-block:: bash | ||
| $ cmake -B build | ||
| $ cd build | ||
| $ make | ||
| Diagnostics | ||
| Running PSC | ||
| =========== | ||
|
|
||
| See the `spack docs <https://spack.readthedocs.io/en/latest/>`_ for how to use spack. Useful commands include ``spack spec``, ``spack compiler list``, and ``spack find``. The ``spack --version`` used here was 0.16.1. | ||
|
|
||
| Use slurm to run PSC. For example, run | ||
|
|
||
| Building | ||
| ======== | ||
|
|
||
| In ``psc``, run | ||
|
|
||
| .. code-block:: sh | ||
| $ cmake -B build | ||
| .. code-block:: bash | ||
| $ sbatch run.sh | ||
| Make sure it builds with gcc 10.2. If it fails, most or all of the following may need to be run (or installed, e.g. ``spack install git``). This may not be an exhaustive list. If CMake fails to find anything, identify the relevant package with ``spack find`` and load them as below. | ||
| where ``run.sh`` looks like | ||
|
|
||
| .. code-block:: sh | ||
| .. code-block:: bash | ||
| #!/bin/bash | ||
| #SBATCH --nodes=4 | ||
| #SBATCH --ntasks-per-node=64 | ||
| #SBATCH --cpus-per-task=1 | ||
| #SBATCH --time=24:00:00 | ||
| $ spack load cmake | ||
| $ spack load git | ||
| $ spack load openmpi | ||
| $ spack load adios2 | ||
| $ spack load hdf5 | ||
| $ spack load gcc | ||
| cp path/to/psc/bits/adios2cfg.xml adios2cfg.xml | ||
| srun path/to/psc/build/src/executable |