-
Notifications
You must be signed in to change notification settings - Fork 9
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Docs: Add compile-notes for cuda on frontera
- Loading branch information
Showing
4 changed files
with
90 additions
and
71 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,67 +1,86 @@ | ||
| # Install CarpetX with Spack (Frontera) | ||
|
|
||
| Use interactive session | ||
| * Use interactive session | ||
|
|
||
| * Compile CPU version: `idev -m 120` | ||
| - Compile CPU version: `idev -m 120` | ||
|
|
||
| * Compile GPU version: `idev -p rtx-dev -m 120` | ||
| - Compile GPU version: `idev -p rtx-dev -m 120` | ||
|
|
||
| Download spack | ||
| * Download spack | ||
|
|
||
| * `git clone -c feature.manyFiles=true https://github.com/spack/spack.git` | ||
| - `git clone -c feature.manyFiles=true https://github.com/spack/spack.git` | ||
|
|
||
| - `git checkout relesases/v0.20` | ||
|
|
||
| * `git checkout relesases/v0.20` | ||
| * Download CarpetX and SpacetimeX | ||
|
|
||
| ## Download CarpetX and SpacetimeX | ||
| ```bash | ||
| curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/master/GetComponents | ||
| chmod a+x GetComponents | ||
| ./GetComponents --root Cactus --parallel --no-shallow https://raw.githubusercontent.com/lwJi/SpacetimeX/main/scripts/spacetimex.th | ||
| ``` | ||
|
|
||
| ``` | ||
| curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/master/GetComponents | ||
| chmod a+x GetComponents | ||
| ./GetComponents --root Cactus --parallel --no-shallow https://raw.githubusercontent.com/lwJi/SpacetimeX/main/scripts/spacetimex.th | ||
| ``` | ||
| ## Intel-Oneapi version (`[email protected]`) | ||
|
|
||
| ``` | ||
| cd Cactus/repos | ||
| git clone https://github.com/lwJi/SpacetimeX.git | ||
| cd ../arrangements | ||
| ln -s ../repos/SpacetimeX | ||
| ``` | ||
| * Load `intel/23.1.0` | ||
|
|
||
| ## Intel-Oneapi version (`oneapi@2023.1.0`) | ||
| - `module load intel/23.1.0` | ||
|
|
||
| Load `intel/23.1.0` | ||
| * Setup spack | ||
|
|
||
| * `module load intel/23.1.0` | ||
| - `. share/spack/setup-env.sh` | ||
|
|
||
| - `spack compiler find` | ||
|
|
||
| Setup spack | ||
| * Create a dir where you want put `view` in (say `/work2/.../username/frontera/SpackView/oneapi`) | ||
|
|
||
| * `. share/spack/setup-env.sh` | ||
| * Replace the last line of `oneapi-23.1.0/spack.yaml` with your own dir (say `/work2/.../username/frontera/SpackView/oneapi/view`) | ||
|
|
||
| * `spack compiler find` | ||
| * Replace the dir `/work2/08708/liwei/frontera/SpackView/oneapi/view` in `configs/config_frontera_oneapi.cfg` (with say `/work2/.../username/frontera/SpackView/oneapi/view`) | ||
|
|
||
| Create a dir where you want put `view` in (say `/work2/.../username/frontera/SpackView/oneapi`) | ||
| * Install other required packages | ||
|
|
||
| * replace the last line of `oneapi-23.1.0/spack.yaml` with your dir (say `/work2/.../username/frontera/SpackView/oneapi/view`) | ||
| ```bash | ||
| env TMPDIR=$WORK/tmp spack --env-dir ./oneapi-23.1.0 compiler find | ||
| env TMPDIR=$WORK/tmp spack --env-dir ./oneapi-23.1.0 concretize --force | ||
| env TMPDIR=$WORK/tmp spack --env-dir ./oneapi-23.1.0 install --fail-fast | ||
| ``` | ||
|
|
||
| * replace the dir `/work2/08708/liwei/frontera/SpackView/oneapi/view` (with say `/work2/.../username/frontera/SpackView/oneapi/view`) | ||
| in `config_frontera_oneapi.cfg` | ||
| * Install SpacetimeX | ||
|
|
||
| Install other required packages | ||
| ```bash | ||
| cd Cactus | ||
| gmake SpacetimeX-oneapi options=repos/SpacetimeX/Docs/compile-notes/frontera/configs/config_frontera_oneapi.cfg | ||
| cp repos/SpacetimeX/Docs/thornlist/spactimex.th configs/SpacetimeX-oneapi/ThornList | ||
| gmake -j24 SpacetimeX-oneapi | ||
| ``` | ||
|
|
||
| * `env TMPDIR=$WORK/tmp spack --env-dir ./oneapi-23.1.0 compiler find` | ||
|
|
||
| * `env TMPDIR=$WORK/tmp spack --env-dir ./oneapi-23.1.0 concretize --force` | ||
| ## CUDA version (`[email protected]`) | ||
|
|
||
| * `env TMPDIR=$WORK/tmp spack --env-dir ./oneapi-23.1.0 install --fail-fast` | ||
| * Make sure rerun `spack install [email protected] %[email protected]` again on `rtx-dev` or `rtx` | ||
|
|
||
| Install CarpetX | ||
| * Create a dir where you want put `view` in (say `/work2/.../username/frontera/SpackView/cuda`) | ||
|
|
||
| * `cd Cactus` | ||
| * Replace the last line of `cuda-11.8.0/spack_yaml` with your dir (say `/work2/.../username/frontera/SpackView/cuda/view`) | ||
|
|
||
| * `gmake CarpetX-oneapi options=config_frontera_oneapi.cfg` | ||
| * Replace the dir `/work2/08708/liwei/frontera/SpackView/cuda/view` in `configs/config_frontera_cuda.cfg` (with say `/work2/.../username/frontera/SpackView/cuda/view`) | ||
|
|
||
| * `cp repos/AsterX/scripts/asterx.th configs/CarpetX-oneapi/ThornList` | ||
| * Install other required packages | ||
|
|
||
| * `gmake -j16 CarpetX-oneapi` | ||
| ```bash | ||
| env TMPDIR=$WORK/tmp spack --env-dir ./cuda-11.8.0 compiler find | ||
| env TMPDIR=$WORK/tmp spack --env-dir ./cuda-11.8.0 concretize --force | ||
| env TMPDIR=$WORK/tmp spack --env-dir ./cuda-11.8.0 install --fail-fast | ||
| ``` | ||
|
|
||
| * Install SpacetimeX | ||
|
|
||
| ```bash | ||
| spack load [email protected] | ||
| spack load [email protected] | ||
| cd Cactus | ||
| gmake SpacetimeX-cuda options=repos/SpacetimeX/Docs/compile-notes/frontera/configs/config_frontera_cuda.cfg | ||
| cp repos/SpacetimeX/Docs/thornlist/spactimex.th configs/SpacetimeX-cuda/ThornList | ||
| gmake -j16 SpacetimeX-cuda | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,47 +19,47 @@ spack: | |
| - [email protected] +hdf5 | ||
| - yaml-cpp | ||
| - zlib | ||
| - gsl | ||
| # - petsc @3.17 +cuda +fftw +hwloc +openmp | ||
| # - gsl | ||
| # - openblas | ||
| packages: | ||
| #petsc: | ||
| # buildable: false | ||
| # externals: | ||
| # - spec: [email protected] | ||
| # prefix: /home1/apps/intel19/impi19_0/petsc/3.17 | ||
| hwloc: | ||
| buildable: false | ||
| externals: | ||
| - spec: [email protected] | ||
| prefix: /opt/apps/hwloc/1.11.12 | ||
| #cmake: | ||
| # buildable: false | ||
| # externals: | ||
| # - spec: [email protected] | ||
| # prefix: /opt/apps/cmake/3.24.2 | ||
| intel-mkl: | ||
| buildable: false | ||
| externals: | ||
| - spec: [email protected] | ||
| prefix: /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl | ||
| fftw: | ||
| buildable: false | ||
| externals: | ||
| - spec: [email protected] | ||
| prefix: /opt/apps/intel19/impi19_0/fftw3/3.3.10 | ||
| - spec: [email protected] | ||
| prefix: /opt/intel/oneapi/mkl/2023.1.0 | ||
| libfabric: | ||
| buildable: false | ||
| externals: | ||
| - spec: libfabric@19.0.9 | ||
| prefix: /opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/libfabric/lib | ||
| intel-mpi: | ||
| - spec: libfabric@2021.9.0 | ||
| prefix: /opt/intel/oneapi/mpi/2021.9.0/libfabric/lib | ||
| intel-oneapi-mpi: | ||
| buildable: false | ||
| externals: | ||
| - spec: intel-mpi@19.0.9 | ||
| prefix: /opt/intel/compilers_and_libraries_2020.4.304/linux/mpi | ||
| - spec: intel-oneapi-mpi@2021.9.0 | ||
| prefix: /opt/intel/oneapi | ||
| mpi: | ||
| buildable: false | ||
| #petsc: | ||
| # buildable: false | ||
| # externals: | ||
| # - spec: [email protected] | ||
| # prefix: /home1/apps/intel19/impi19_0/petsc/3.17 | ||
| #cmake: | ||
| # buildable: false | ||
| # externals: | ||
| # - spec: [email protected] | ||
| # prefix: /opt/apps/cmake/3.24.2 | ||
| #fftw: | ||
| # buildable: false | ||
| # externals: | ||
| # - spec: [email protected] | ||
| # prefix: /opt/apps/intel19/impi19_0/fftw3/3.3.10 | ||
| specs: | ||
| - matrix: | ||
| - [$%compilers] | ||
|
|
||