MOOSE on Leonardi@UNSW

Leonardi is a HPC cluster from the Faculty of Engineering at UNSW. This page documents the general commands on the cluster and how to run REDBACK/MOOSE on Leonardi.

All necessary files can be found in the folder scripts/leonardi.

Leonardi General Usage

Connection to the cluster

1. From your terminal, type ssh [email protected] (replace by your zID)
2. You will be prompted to enter a password which is your zpass.

For Windows machines, use PUTTY.

Modules

• Access to available modules: module avail
• Access to currently loaded modules: module list
• Loading module X: load module X
• Unloading module X: unload module X
• Unloading all modules: module purge

Submitting a job

sbatch run_job.sh

Tracking Jobs

• Status of jobs: squeue –u z1234567 (your zID)

You will see lines like this:

JOBID  PARTITION  NAME   USER     ST    TIME       NODES NODELIST(REASON)
26062   short64   JOB   z1234567   R    5:08:11      1      ec06b01

R means the job is running.

• To watch a job's output in real-time: tail -f logs/run-26062.log (filename of output)
• To see when your job will be started: squeue --start -j 26062 (job ID)
• To cancel a job: scancel 26062 (job ID)
• For job optimization, the following may be useful (to be expanded on): Slurm srun Sacct Scontrol show job

How to transfer files To/From Leonardi

We use FileZilla (be careful of the bundled adware! Don't use the Sourceforge installer).

Do NOT use FireFTP. It causes file corruption.

MOOSE Structure on Leonardi

For a more user-friendly use of Leonardi, we decided to install one common MOOSE version. Users will not have to install it on their account, they will only need to install their own REDBACK (that will be built on this MOOSE).

This allows the user to have a functional MOOSE at all time and at the same time to have a personal REDBACK version where they can work on their own branch.

MOOSE and its prerequisites are installed in /home/z5014651/opt/moose. Martin Lesueur is the person in charge of this account. If you need any update on MOOSE, you need to contact him.

For a personal installation of MOOSE, please refer to leonardi_build.sh.

Installing REDBACK

1. Transfer leonardi_redback_build.sh in your home directory on Leonardi.
2. Edit this file, change BRANCH_NAME on line 13 to the name of your branch on REDBACK
3. chmod u+x leonardi_redback_build.sh to make that file executable
4. dos2unix leonardi_redback_build.sh to remove trailing white spaces
5. ./ leonardi_redback_build.sh to run the script. You will see the output of the commands being executed.

You need to make sure that all the tests of REDBACK are passing!

Note that you can’t use Leonardi during this installation and MUST not log out!

Running MOOSE or REDBACK

We advise you to create directories

• to store jobs output: mkdir $HOME/logs
• to store input files, output files and meshes: mkdir $HOME/run_files

In run_job.sh, edit

• Partition, this is the name of the queue you will submit your job into. All available queues are documented in Leonardi’s wiki.
• Job’s allocated resources: walltime, number of nodes, number of processors (ntasks). These attributes often depend on the partition chosen.
• the executable
  • $HOME/projects/redback/redback-opt to use REDBACK
  • /home/z5014651/opt/moose/moose/modules/combined/modules-opt to use MOOSE modules
• the executed command line: mpirun -N 47 -n 47 $EXECUTABLE -i input_file.i
  • N is the number of processors per node
  • n is the total number of processors

You can add more options at the end of the command line that allows you to personalise the submitted input file like these two:

• Mesh/file={mesh_location} to change the mesh you run the simulation on
• Outputs/file_base={output_filebase} to change the output name and location of the simulatin

Further documentation

Leonardi website has a tutorial introducing you to Linux and HPC in general.

Refer to the slurm documentation for more sbatch commands to run the job.