INSTALL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Rayleigh Installation Instructions

Throughout this Document, <rayleigh root> refers to the directory in which this
file (named 'INSTALL') is located.  If this file is not located in <rayleigh
root>, then you should identify the location of <rayleigh root> before
proceeding.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I.  Overview of the Installation Process

Compilation and installation of Rayleigh is accomplished using the "configure /
make / make install " process.  Building Rayleigh requires the execution of
three commands from within <rayleigh root>:

1. ./configure --OPTION1=... -OPTION2=... 
2. make
3. make install

(4.) Running 'make distclean' will revert <rayleigh root> to it's
pre-configured state.

The configure script automatically supports the GNU, Intel, AOCC, and Cray
compilers.  An IBM-compiler example is provided at the end of this document.
On many Intel-based machines, running ./configure or ./configure -mkl  will
suffice.  In other circumstances, it may be necessary to specify a subset of
configure's options.  Some possible variations are illustrated in section III.

To see a detailed description of all available options, run ./configure -h.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

II.  Dependencies

Rayleigh requires the the following compilers and libraries:

1. Fortran-MPI compiler that supports the Fortran 2003 standard.
2. C++ compiler (does not need to be MPI-capable)

3. Intel's Math Kernel Library (MKL)

    OR

3a. BLAS
3b. LAPack
3c. FFTW (version 3.x)


MKL provides an interface to BLAS, LAPack, and FFTW routines.   
We strongly recommend that you link against the MKL library
when running Rayleigh on Intel processors.  This is true even if
you are not compiling with the Intel compiler.

If you do not have MKL installed, then you will need to know the location
of your BLAS, LAPACK, and FFTW installations.  In particular, the configure
script requires that you know the "root" directory for these installations.
By "root," we mean the directory that contains subdirectories named "lib,"
"include," etc.   Rayleigh expects to find the following four files when
building without MKL:

<BLAS ROOT>/lib/libblas.so  OR <BLAS ROOT>/lib/libopenblas.so
<LAPACK ROOT>/lib/liblapack.so
<FFTW ROOT>/lib/libfftw3.so
<FFTW ROOT>/include/fftw3.f

These libraries may be linked statically, as discussed in section III.
In that case, similarly named library files with a '.a' extension will
be required.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
III.  Configuration

III.a  Overview

Rayleigh's configure script currently supports the Intel and GNU compilers.  
For other compilers, the optimization, include, and library  flags 
will need to be set manually by passing approriate flags to configure.

NOTE:  If your system uses a module system to manage software, be sure to load 
all relevant modules (e.g., intel, impi, mkl, etc.) before running 
configure.

The configure process also determines where the rayleigh executables will 
be placed. By default, executables are placed in <rayleigh root>/bin. 

The 'bin' subdirectory is created when "make install" is run.  If you 
would like to install Rayleigh to another directory, use the prefix 
keyword when running configure.  E.g.,

./configure --prefix=<rayleigh install>

The compilation process generates two executables:  
    <rayleigh install>/bin/rayleigh.opt 
        and 
    <rayleigh install>/bin/rayleigh.dbg

The rayleigh.opt executable is compiled with optimization flags appropriate
for your compiler type and should be used for production runs.

The rayleigh.dbg executable is compiled with debugging flags and should
not be used for production runs.  Instead, if Rayleigh crashes unexpectedly,
it is a good idea to re-run the same job with rayleigh.dbg and report
the error message(s) to the development team.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
III.b   Configuration using the Intel compiler and MKL.

Running ./configure will work "out of the box" on many Intel systems, 
provided the following environment variables are defined:

1. FC  (the name of your Fortran-MPI compiler; e.g., mpiifort, mpif90 etc.)
2. CC  (the name of your C-compiler; e.g., icc or icpc)
3. MKLROOT (the location of your MKL installation)

If these environment variables are undefined, or if they are set to something 
that you do not wish to override, invoke configure in a fashion similar to:

./configure --with-mkl=<MKL root> --CC=icc --FC=mpiifort (or mpifc, mpif90, etc.)

NOTE:  When compiling with Intel, you will be prompted to specify the level of 
vectorization. Alternatively, you may compile for all levels of vectorization 
supported by Intel processors. If you choose to do so, no rayleigh.opt will be
produced.  Instead, rayleigh.avx, rayleigh.sse, and rayleigh.avx2 will be
generated (in addition to rayleigh.dbg).

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
III.c Configuration using the GNU compiler and MKL

MKL can also be linked to code compiled using GFortran.  The configuration 
process in this situation is similar to that used for Intel compilers, but 
configure is called with the -mkl flag. If the MKLROOT, FC, and CC environment 
variables are set, run the following:

./configure -mkl

If they are not set, run:

./configure --with-mkl=<mkl root> --CC=gcc --FC=mpif90 (or mpifc, etc.)

As with Intel compilation, FC should refer to your Fortran-MPI compiler.
Be sure to set FC to mpif90 or mpifc, and not to gfortran.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
III.d  Configuration using non-MKL libraries

Rayleigh can be configured to use your preferred FFTW, BLAS, and LAPACK
libraries.  These directories are specified using the --with-blas,
--with-lapack, and --with-fftw flags.   By default, Raleigh links dynamically
to shared libraries.  To alter this behavior, use the  --static-blas,
--static-lapack, and or --static-fftw flags.  If using OpenBLAS instead of
BLAS, run configure with the -openblas flag.

The following example illustrates how to link with shared OpenBLAS and FFTW libraries,
and a static LAPack library:

BLASROOT=/software/openblas/0.2.20
LAPACKROOT=/software/lapack/3.8.0
FFTWROOT=/software/fftw/3.3.4
./configure \
--FC=mpif90 \
--CC=gcc \
--with-blas=$BLASROOT \
--with-lapack=$LAPACKROOT \
--with-fftw=$FFTWROOT \
--static-lapack \
--openblas

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
III.e  Configuration using Alternative Compilers / Customized Compiler Flags

For compilers other than GNU, Intel, AOCC, or Cray, you will need to manually
specify the optimization, library, and include flags.  You may also wish to do
this if you are experimenting with new compilation flags that are not
automatically provided by configure for known compilers.   These flags can be set
using --LIBFLAGS, --FFLAGS_OPT, --FFLAGS_DBG, and --INCLUDE.  FFLAGS_OPT and
FFLAGS_DBG define the optimization flags used for rayleigh.opt and rayleigh.dbg
respectively.

Alternatively, the --FFLAGS can be set to use the same optimization flags for
rayleigh.opt and rayleigh.dbg.


NOTE:  the -nodirs (or --nodirs) flag can be set on Machines where linking
C-code code to Fortran code via Rayleigh's Makefile produces errors.  In that
case, users will need to create their subdirectories for each Rayleigh run
manually.  This can be done by running the script rayleigh/etc/make_dirs from
within a run directory.

The example below shows how to configure the code for Mira, an IBM Blue Gene/Q.

export BLAS_LIB=/soft/libraries/alcf/current/xl/BLAS/lib
export LAPACK_LIB=/soft/libraries/alcf/current/xl/LAPACK/lib
export FFTW_LIB=/soft/libraries/alcf/current/xl/FFTW3/lib
export FFTW_INC=/soft/libraries/alcf/current/xl/FFTW3/include

./configure \
--FC=mpixlf2003_r \
--CC=gcc \
--LIBFLAGS="-L$FFTW_LIB -lfftw3 -L$LAPACK_LIB -llapack -L$BLAS_LIB -lblas" \
--INCLUDE="-I . -I$FFTW_INC" \
--FFLAGS_OPT="-qfree=f90 -O2" \
--FFLAGS_DBG="-qfree=f90 -O0 -g -traceback  -qcheck  -C" \
--nodirs

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
III.f  Configuration on a Mac using brew package manager

Perhaps the simplest way to compile Rayleigh on a Mac is to begin by installing
the required compilers and libraries with Homebrew. Homebrew installs all the
stuff you need that Apple didn't. To install Homebrew on your system, follow
the instructions on this website: https://brew.sh/

Next, install the GNU, MPI, and FFTW compilers and libraries with Homebrew as
below.

brew install gcc open-mpi fftw

Working in a bash shell, the example below shows how to configure the code. To
install Rayleigh, change the <install prefix> to an appropriate location on
your machine.

export FC=mpif90
export CC=gcc
export FFTWROOT=/usr/local
export LAPACKROOT=/usr
export BLASROOT=/usr
export PREFIX=<installation prefix>

./configure --FC=$FC --CC=$CC --with-fftw=$FFTWROOT --with-lapack=$LAPACKROOT --with-blas=$BLASROOT --prefix=$PREFIX
make
make install

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
IV

Ultimately, the role of 'configure' is to generate a definitions file that is
included by 'Make' at compilation time.  This file is named
"Machine_Definitions," and it is placed in <rayleigh root>/src/build by
configure.  As a last resort, you may edit this file directly in order to
customize the compiler flags generated by running configure.  If the configure
process fails to create Machine_Definitions, your build is a challening one.
Run './configure --blank' to generate a template that you may edit as needed.
IMPORTANT:  Ensure that whitespace is removed at the end of each line in this
file.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
V Building the documentation

If you plan to modify the documentation you may want to build it locally to
check your changes before submission. To do so you need a number of additional
dependencies that are not required to build the Rayleigh executable. In
particular you need the following packages:

sphinx (including its dependencies)
pandoc
python
numpy
scipy
sphinxcontrib-bibtex
nbsphinx
jupyter-client
ipykernel

In order to build the documentation navigate to the doc/ directory, and start
the sphinx build with 'make html' to build the html documentation, or 'make
latexpdf' to build the pdf manual. The generated files will be placed in
'doc/build/html' or 'doc/build/latex' respectively. Note that the design of the
html documentation will deviate slightly from the online version.