MOOSE on Raijin@NCI

(This page is under construction).

Modules

Raijin has most of the software we'd want already installed, as modules:

module load intel-cc/17.0.1.132
module load intel-mpi/2017.0.098
module load vtk/6.3.0
module load cmake/3.6.2

Queues

For each Sandy Bridge node, we have 2(cpu) x 8(cores) x 2(hyperthreading) = 32 threads.

normal: largest number of requestable CPUs.  Processor-hours cost 1 credit each.
express: shorter wait time.  Processor-hours cost 3 credits each.
copyq: the only queue that has internet access.  1 CPU and walltime <= 10 hours.  1 credit/processor-hour.

There are other specialized queues; see https://opus.nci.org.au/display/Help/Raijin+User+Guide

Handy commands:

qsub <job_file> : submit job
qstat -u <username> : show job stats.
qdel <job_number> : delete job from queue
lquota : see disk quotas
nci_account : see group hours usage

The home directory has a tiny 2GB quota. However, there is a shared drive under /short/ws55 with 72GB quota. There is also a 1TB massdata quota, which requires special commands and is only to be used to archive large data files (not very useful) https://opus.nci.org.au/display/Help/MASSDATA+User+Guide.

Petsc

Raijin's Petsc module lacks Hypre; Libmesh complains about this and doesn't compile. I'm currently trying to build a working install. I'll see if I can make a shared installation. I'm using the Intel compilers; They're probably a bit faster since the CPUs are Sandy Bridge/Broadwell-E.

Anaconda

Add this to your .bashrc:

export PATH="/home/561/jl1719/anaconda2/bin:$PATH"

remember to run

dos2unix .bashrc

Anaconda is ~2GB. If this works as I expect, it would be better than having separate installs, since there's a tiny disk quota.