INSTALL

                    Finite Element Discretization Library
                                   __
                       _ __ ___   / _|  ___  _ __ ___
                      | '_ ` _ \ | |_  / _ \| '_ ` _ \
                      | | | | | ||  _||  __/| | | | | |
                      |_| |_| |_||_|   \___||_| |_| |_|

                               https://mfem.org

The MFEM library has a serial and an MPI-based parallel version, which largely
share the same code base. The only prerequisite for building the serial version
of MFEM is a (modern) C++ compiler, such as g++. The parallel version of MFEM
requires an MPI C++ compiler, as well as the following external libraries:

- hypre (a library of high-performance preconditioners)
  https://github.com/hypre-space/hypre

- METIS (a family of multilevel partitioning algorithms)
  http://glaros.dtc.umn.edu/gkhome/metis/metis/overview

The hypre dependency can be downloaded as a tarball from GitHub or from the
project webpage https://www.llnl.gov/casc/hypre. For example, the 2.20.0 release
of hypre is available at

   https://github.com/hypre-space/hypre/archive/v2.20.0.tar.gz

The METIS dependency can be disabled but that is not generally recommended, see
the option MFEM_USE_METIS.

MFEM also includes support for devices such as GPUs, and programming models such
as CUDA, HIP, OCCA, OpenMP and RAJA.

- Starting with version 4.0, MFEM requires a C++11 compiler. We recommend using
  a newer compiler, e.g. GCC version 4.9 or higher.

- CUDA support requires an NVIDIA GPU and an installation of the CUDA Toolkit
  https://developer.nvidia.com/cuda-toolkit

- HIP support requires an AMD GPU and an installation of the ROCm software stack
  https://rocm.github.io/ROCmInstall.html#installing-from-amd-rocm-repositories

- OCCA support requires the OCCA library
  https://libocca.org

- OpenMP support requires a compiler implementing the OpenMP API
  https://www.openmp.org

- RAJA support requires installation of the RAJA performance portability layer
  with (optionally) support for CUDA and OpenMP
  https://github.com/LLNL/RAJA

The library supports two build systems: one based on GNU make, and a second one
based on CMake. Both build systems are described below. Some hints for building
without GNU make or CMake can be found at the end of this file.

In addition to the native build systems, MFEM packages are also available in the
following package managers:

- Spack, https://github.com/spack/spack
- OpenHPC, http://openhpc.community
- Homebrew/Science, https://github.com/Homebrew/homebrew-science (deprecated)

We also recommend downloading and building the MFEM-based GLVis visualization
tool which can be used to visualize the meshes and solution in MFEM's examples
and miniapps. See https://glvis.org and https://mfem.org/building.

Quick start with GNU make
=========================
Serial build:
   make serial -j 4

Parallel build:
   (download hypre and METIS 4 from above URLs)
   (build METIS 4 in ../metis-4.0 relative to mfem/)
   (build hypre in ../hypre relative to mfem/)
   make parallel -j 4

CUDA build:
   make cuda -j 4
   (build for a specific compute capability: 'make cuda -j 4 CUDA_ARCH=sm_30')

HIP build:
   make hip -j 4
   (build for a specific AMD GPU chip: 'make hip -j 4 HIP_ARCH=gfx900')

Example codes (serial/parallel, depending on the build):
   cd examples
   make -j 4

Build everything (library, examples and miniapps) with current configuration:
   make all -j 4

Quick-check the build by running Example 1/1p (optional):
   make check

Quick start with CMake
======================
Serial build:
   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   cmake <mfem-source-dir>
   make -j 4   (assuming "UNIX Makefiles" generator)

Parallel build:
   (download hypre and METIS 4 from above URLs)
   (build METIS 4 in ../metis-4.0 relative to mfem/)
   (build hypre in ../hypre relative to mfem/)
   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   cmake <mfem-source-dir> -DMFEM_USE_MPI=YES
   make -j 4

CUDA build:
   (this build requires CMake 3.8 or newer)
   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   cmake <mfem-source-dir> -DMFEM_USE_CUDA=YES
   make -j 4

Example codes (serial/parallel, depending on the build):
   make examples -j 4

Build everything (library, examples and miniapps) with current configuration:
   make exec -j 4

Quick-check the build by running Example 1/1p (optional):
   make check


Building with GNU make
======================
The MFEM build system consists of two steps: configuration and compilation.

The configuration step can be used to adjust paths to external libraries,
compilers, flags, etc, similar to "./configure". It is performed by running

   make config [OPTIONS] ...

The OPTIONS are of the form VARIABLE=VALUE. Detailed description of the
configuration options is given below. Alternatively, the options can be
specified with an input file:

   cp config/defaults.mk config/user.mk
   (edit config/user.mk)
   make config

Note that config/user.mk, if present, is loaded after config/defaults.mk and
its path/name can be changed with

   make config USER_CONFIG=<user_config_file>

The build system can be configured to use a separate build directory, for an
out-of-source build. There are two ways to do that: the first one is

   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   make -f <mfem-source-dir>/makefile config [OPTIONS] ...

The second one is

   cd <mfem-source-dir>
   make BUILD_DIR=<mfem-build-dir> config [OPTIONS] ...

Note that in both cases the default location for the (optional) user
configuration file is <mfem-build-dir>/config/user.mk.

Once configured, the library can be built simply with

   cd <mfem-build-dir>   (if building out-of-source)
   make

Note that re-configuration is only needed to change the currently configured
options. Several shortcut targets combining (re-)configuration and compilation
are also defined:

   make serial    -> Builds serial optimized version of the library
   make parallel  -> Builds parallel optimized version of the library
   make debug     -> Builds serial debug version of the library
   make pdebug    -> Builds parallel debug version of the library
   make cuda      -> Builds serial cuda optimized version of the library
   make pcuda     -> Builds parallel cuda optimized version of the library
   make cudebug   -> Builds serial cuda debug version of the library
   make pcudebug  -> Builds parallel cuda debug version of the library
   make hip       -> Builds serial hip optimized version of the library
   make phip      -> Builds parallel hip optimized version of the library
   make hipdebug  -> Builds serial hip debug version of the library
   make phipdebug -> Builds parallel hip debug version of the library

Note that any of the above shortcuts accept configuration options, either at the
command line or through a user configuration file.

The build can be quick-tested by running

   make check

which will simply compile and run Example 1/1p. For more extensive tests that
check the results from all the serial/parallel MFEM examples and miniapps use:

   make test

Note that by default MFEM uses "mpirun -np" in its test runs (this is also what
is used in the sample runs of its examples and miniapps). The MPI launcher can
be changed by the user as described in the "Specifying an MPI job launcher"
section at the end of this file.

Running all the tests may take a while. Implementation details about the check
and test targets can be found in the top-level makefile and the config/test.mk
file.

An optional installation of the library and the headers can be performed with

   make install [PREFIX=<dir>]

The library will be installed in $(PREFIX)/lib, the headers in
$(PREFIX)/include, and the configuration makefile (config.mk) in
$(PREFIX)/share/mfem. The PREFIX option can also be set during configuration.

Information about the current build configuration can be viewed using

   make status
   make info

To clean the library and object files, but keep the current configuration, use

   make clean

To clean everything, including the current configuration, use

   make distclean

For a short help message, use

   make help

The build process creates the MFEM library (libmfem.a) and the include file
(mfem.hpp) needed in MFEM-based applications, see e.g. the example codes in the
examples/ directory or the miniapps in the miniapps/ directory. A selected
subset of configuration options and derived makefile variables are also written
to the file config/config.mk. This file can be included by other makefiles to
obtain information about the MFEM configuration, see e.g. the makefile in the
examples/ directory.


Configuration options (GNU make)
================================
See the configuration file config/defaults.mk for the default settings.

Compilers:
   CXX      - C++ compiler, serial build
   MPICXX   - MPI C++ compiler, parallel build
   CUDA_CXX - The CUDA compiler, 'nvcc'

Compiler options:
   OPTIM_FLAGS - Options for optimized build
   DEBUG_FLAGS - Options for debug build
   CXXFLAGS    - If not set, defined based on the above optimized/debug flags
   CPPFLAGS    - Additional compiler options

Build options:
   STATIC - Build a static version of the library (YES/NO), default = YES
   SHARED - Build a shared version of the library (YES/NO), default = NO

Installation options:
   PREFIX  - Specify the installation directory. The library (libmfem.a) will be
             installed in $(PREFIX)/lib, the headers in $(PREFIX)/include, and
             the configuration makefile (config.mk) in $(PREFIX)/share/mfem.
   INSTALL - Specify the install program, e.g /usr/bin/install

MFEM library features/options (GNU make)
----------------------------------------
MFEM_USE_MPI = YES/NO
   Choose parallel/serial build. The parallel build requires proper setup of the
   HYPRE_* and METIS_* library options, see below.

MFEM_USE_METIS = YES/NO
   Enable/disable the use of the METIS library. By default, this option is set
   to the value of MFEM_USE_MPI. If this option is explicitly disabled in a
   parallel build, then the only parallel partitioning (domain decomposition)
   option in the library will be Cartesian partitioning with box meshes, and
   thus most of the parallel examples and miniapps will fail.

MFEM_DEBUG = YES/NO
   Choose debug/optimized build. The debug build enables a number of messages
   and consistency checks that may simplify bug-hunting.

MFEM_USE_EXCEPTIONS = YES/NO
   Enable the use of exceptions. In particular, modifies the default behavior
   when errors are encountered: throw an exception, instead of aborting.

MFEM_USE_LIBUNWIND = YES/NO
   Use libunwind to print a stacktrace whenever mfem_error is raised. The
   information printed is enough to determine the line numbers where the
   error originated, provided MFEM_DEBUG=YES or build flags include `-g'.

MFEM_USE_METIS_5 = YES/NO
   Specify the version of the METIS library - 5 (YES) or 4 (NO).

MFEM_USE_LAPACK = YES/NO
   Use LAPACK routines for various dense linear algebra operations. When
   enabled, this option uses the LAPACK_* library options, see below. (When not
   enabled MFEM provides simple internal implementations where appropriate.)

MFEM_THREAD_SAFE = YES/NO
   Use thread-safe implementation for some classes/methods. This comes at the
   cost of extra memory allocation and de-allocation.

MFEM_USE_LEGACY_OPENMP = YES/NO
   Enable (basic) experimental OpenMP support. Requires MFEM_THREAD_SAFE.
   This option is deprecated.

MFEM_USE_OPENMP = YES/NO
   Enable the OpenMP backend.

MFEM_USE_MEMALLOC = YES/NO
   Internal MFEM option: enable batch allocation for some small objects.
   Recommended value is YES.

MFEM_TIMER_TYPE = 0/1/2/3/4/5/6/NO
   Specify which library functions to use in the class StopWatch used for
   measuring time. The available options are:
      0  - use std::clock from <ctime>, standard C++
      1  - use times from <sys/times.h>
      2  - use high-resolution POSIX clocks (see option POSIX_CLOCKS_LIB)
      3  - use QueryPerformanceCounter from <windows.h>
      4  - use mach_absolute_time from <mach/mach_time.h> + std::clock (Mac)
      5  - use gettimeofday from <sys/time.h>
      6  - use MPI_Wtime from <mpi.h>
      NO - use option 3 if the compiler macro _WIN32 is defined, 0 otherwise

MFEM_USE_SUNDIALS = YES/NO
   Enable MFEM time integrators and non-linear solvers based on the SUNDIALS
   library. When enabled, this option uses the SUNDIALS_* library options,
   see below.

MFEM_USE_MESQUITE = YES/NO
   Enable MFEM functionality based on the Mesquite library. When enabled, this
   option uses the MESQUITE_* library options, see below.

MFEM_USE_SUITESPARSE = YES/NO
   Enable MFEM functionality based on the SuiteSparse library. Currently, this
   option adds the classes UMFPackSolver and KLUSolver (both sparse serial
   direct solvers). When enabled, this option uses the SUITESPARSE_* library
   options, see below.

MFEM_USE_SUPERLU = YES/NO
   Enable MFEM functionality based on the SuperLU_DIST library. Currently, this
   option adds the classes SuperLUSolver (a parallel sparse direct solver) and
   SuperLURowLocMatrix a distributed CSR matrix class needed by SuperLU. When
   enabled, this option uses the SUPERLU_* library options, see below.

MFEM_USE_SUPERLU5 = YES/NO
   If SuperLU functionality is enabled, use the older 5.1.0 version rather than
   the more recent 6+ versions.

MFEM_USE_MUMPS = YES/NO
   Enable MFEM functionality based on the MUMPS library. Currently, this
   option adds the class MUMPSSolver (a parallel sparse direct solver). 
   When enabled, this option uses the MUMPS_* library options, see
   below.
   
MFEM_USE_STRUMPACK = YES/NO
   Enable MFEM functionality based on the STRUMPACK sparse direct solver and
   preconditioner through the STRUMPACKSolver and STRUMPACKRowLocMatrix
   classes. When enabled, this option uses the STRUMPACK_* library options, see
   below.

MFEM_USE_GINKGO = YES/NO
   Enable MFEM functionality based on the Ginkgo library, which provides
   iterative linear solvers and preconditioners with OpenMP, CUDA backends, see
   https://github.com/ginkgo-project/ginkgo. When enabled, the user can use
   Ginkgo's solvers and preconditioners as shown in examples/ginkgo/.

MFEM_USE_AMGX = YES/NO
   Enable MFEM functionality based on the AmgX multigrid library from NVIDIA.
   Allows the user to use SparseMatrices and HypreParMatrices to solve linear
   systems with the routines from the AmgX library.

MFEM_USE_GNUTLS = YES/NO
   Enable secure socket support in class socketstream, using the auxiliary
   GnuTLS_* classes, based on the GnuTLS library. This option may be useful in
   multi-user environment to prevent users from sending/receiving visualization
   data to/from other users. When this option is enabled, the default behavior
   in class socketstream is to use secure sockets, e.g. when connecting to a
   GLVis visualization server. In order for this to work, one needs to generate
   GLVis server/client key pairs (in ~/.config/glvis), similar to ssh keys --
   the script 'glvis-keygen.sh' in the main GLVis directory can be used to do
   that:
      bash glvis-keygen.sh ["Your Name"] ["Your Email"]
   In MFEM v3.3.2 and earlier, the secure authentication is based on OpenPGP
   keys, while later versions use X.509 certificates. The latest version of the
   script 'glvis-keygen.sh' can be used to generate both types of keys.
   When MFEM_USE_GNUTLS is enabled, the additional build options, GNUTLS_*, are
   also used, see below.

MFEM_USE_NETCDF = YES/NO
   NetCDF is the library that is used by the SNL Cubit mesh generator to create
   Genesis mesh files. This option enables a reader for these files, which
   requires that NetCDF be installed, see the NETCDF_* build options below.

MFEM_USE_PETSC = YES/NO
   Enable MFEM linear and non-linear solvers, preconditioners, time integrators
   and other features based on the PETSc package. When enabled, this option uses
   the PETSC_* library options, see below.

MFEM_USE_SLEPC = YES/NO
   Enable MFEM eigensolvers based on the SLEPc package. When enabled, this
   option uses the SLEPC_* library options, see below.

MFEM_USE_MPFR = YES/NO
   MPFR is a library for multiple-precision floating-point computations. This
   option enables the use of MPFR in MFEM, e.g. for precise computation of 1D
   quadrature rules. When enabled, this option uses the MPFR_* library options,
   see below.

MFEM_USE_SIDRE = YES/NO
   Sidre is a component of LLNL's axom project, https://github.com/LLNL/axom,
   that provides an HDF5-based file format for visualization or restart
   capability following the Conduit (https://github.com/LLNL/conduit) mesh
   blueprint specification. When enabled, this option requires installation of
   HDF5 (see also MFEM_USE_NETCDF), Conduit and LLNL's axom project.

MFEM_USE_SIMD = YES/NO
   Enables the high performance templated classes to use architecture dependent
   SIMD intrinsics instead of the generic implementation of class AutoSIMD in
   linalg/simd/auto.hpp. This option should be combined with suitable
   compiler options, such as -march=native, to enable optimal vectorization.

MFEM_USE_CONDUIT = YES/NO
   Enables support for converting MFEM Mesh and Grid Function objects to and
   from Conduit Mesh Blueprint Descriptions (https://github.com/LLNL/conduit/)
   and support for JSON and Binary I/O via Conduit Relay. This option requires
   an installation of Conduit. If Conduit was built with HDF5 support, it also
   requires an installation of HDF5 (see also MFEM_USE_NETCDF).

MFEM_USE_ADIOS2 = YES/NO
   Enables support for ADIOS2, version 2 of the adaptable input output system
   for scientific data management. In MFEM, ADIOS2 provides parallel I/O with
   ParaView visualization.

MFEM_USE_ZLIB = YES/NO
   Enables use of on-the-fly gzip compressed streams. With this feature enabled
   (YES), MFEM can compress its output files on-the-fly. In addition, it can
   read back files compressed with zlib (or any compression utility capable
   of creating a gzip-compatible output such as gzip).
   MFEM will write compressed files if the mode argument in the constructor
   includes a 'z' character. With this feature disabled (NO), MFEM will not be
   able to properly read an input file if it is gzip compressed. In that case,
   the solution is to uncompress the file with an external tool (such as gunzip)
   before attempting to use it with MFEM.
   When enabled, this option uses the ZLIB_* library options, see below.

MFEM_USE_PUMI = YES/NO
   Enable the usage of PUMI (https://scorec.rpi.edu/pumi/) in MFEM. The Parallel
   Unstructured Mesh Infrastructure (PUMI) is an unstructured, distributed mesh
   data management system that is capable of handling general non-manifold
   models and effectively supports automated adaptive analysis. PUMI enables
   support for parallel unstructured mesh modifications in MFEM.
   The develop branch of PUMI repository (https://github.com/SCOREC/core)
   should be used for most updated features.

MFEM_USE_UMPIRE = YES/NO
   Enables support for Umpire, a resource management library that allows the
   discovery, provision, and management of memory on machines with multiple
   memory devices like NUMA and GPUs.

MFEM_USE_HIOP = YES/NO
  Enable the usage of HiOp (https://github.com/LLNL/hiop) in MFEM. HiOp is an
  HPC solver for nonlinear optimization problems.

MFEM_USE_CUDA = YES/NO
   Enables support for CUDA devices in MFEM. CUDA is a parallel computing
   platform and programming model for general computing on graphical processing
   units (GPUs). The variable CUDA_ARCH is used to specify the CUDA compute
   capability used during compilation (by default, CUDA_ARCH=sm_60). When
   enabled, this option uses the CUDA_* build options, see below.

MFEM_USE_HIP = YES/NO
   Enables support for AMD devices in MFEM. HIP is a heterogeneous-compute
   interface for portability developed by AMD that can target both AMD and
   NVIDIA GPUs. The variable HIP_ARCH is used to specify the AMD GPU processor
   used during compilation (by default, HIP_ARCH=gfx900).  When enabled, this
   option uses the HIP_* build options, see below.

MFEM_USE_RAJA = YES/NO
   Enable support for the RAJA performance portability layer in MFEM. RAJA
   provides a portable abstraction for loops, supporting different programming
   model backends. When using RAJA built with CUDA support, CUDA support must be
   also enabled in MFEM, i.e. MFEM_USE_CUDA=YES must be set.

MFEM_USE_OCCA = YES/NO
   Enables support for the OCCA library in MFEM. OCCA is an open-source library
   which aims to make it easy to program different types of devices (e.g. CPU,
   GPU, FPGA) by providing an unified API for interacting with JIT-compiled
   backends. In order to use the OCCA CUDA backend, CUDA support must be enabled
   in MFEM as well, i.e. MFEM_USE_CUDA=YES must be set.

MFEM_USE_GSLIB = YES/NO
   Enables MFEM functionality based on the GSLIB library, and specifically its
   FindPoints component, which provides a robust algorithms to evaluate finite
   element functions in a collection of points in physical space. When enabled,
   the user can use the GSLIB-FindPoints methods as shown in miniapps/gslib.

MFEM_USE_CEED = YES/NO
   Enables support for the libCEED library in MFEM. libCEED is a portable
   library for performant high-order operator evaluation developed by the Center
   for Efficient Exascale Discretizations in the Exascale Computing Project.

MFEM_USE_MKL_CPARDISO = YES/NO
   Enables the interface to MKL CPardiso: the Intel MKL Parallel Direct Sparse
   Solver for Clusters. Make sure to set the correct values for MKL_MPI_WRAPPER
   and MKL_LIBRARY_SUBDIR as shown in defaults.mk. If you configure MFEM with
   MFEM_USE_LAPACK=YES, verify that the MKL LAPACK libraries are used. The
   OpenMP capabilities are disabled at link time.

MFEM_BUILD_TAG = (any value)
   An optional tag to characterize the build. Exported to config/config.mk.
   Can be used to identify the MFEM build from other makefiles.

VERBOSE = YES/NO
   Print some informational messages when building.

External libraries (GNU make):
------------------------------
Two types of library configuration options are used:
   <LIBNAME>_OPT - for compiler options which usually specify an include path,
                   e.g.: -I/home/user/hypre/include
   <LIBNAME>_LIB - for link options which usually specify link path and library
                   name, e.g.: -L/home/user/hypre/lib -lHYPRE

If specifying relative paths, they should be relative to the top-level MFEM
directory and use the string @MFEM_DIR@, e.g. HYPRE_OPT = -I@MFEM_DIR@/../hypre.

The specific libraries and their options are:

- HYPRE, required for the parallel build, i.e. when MFEM_USE_MPI = YES.
  URL: https://github.com/hypre-space/hypre and https://www.llnl.gov/casc/hypre
  Options: HYPRE_OPT, HYPRE_LIB.
  Versions: HYPRE >= 2.10.0b.

- METIS, used when MFEM_USE_METIS = YES. If using METIS 5, set
  MFEM_USE_METIS_5 = YES (default is to use METIS 4).
  URL: http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
  Options: METIS_OPT, METIS_LIB.
  Versions: METIS 4.0.3 or 5.1.0.

- LAPACK (optional), used when MFEM_USE_LAPACK = YES. Alternative, optimized
  implementations can also be used, e.g. the ATLAS project.
  URL: http://www.netlib.org/lapack (LAPACK)
       http://math-atlas.sourceforge.net (ATLAS)
  Options: LAPACK_OPT (currently not used/needed), LAPACK_LIB.

- OpenMP (optional), usually part of compiler, used when either MFEM_USE_OPENMP
  or MFEM_USE_LEGACY_OPENMP is set to YES.
  Options: OPENMP_OPT, OPENMP_LIB.

- High-resolution POSIX clocks: when using MFEM_TIMER_TYPE = 2, it may be
  necessary to link with a system library (e.g. librt.so).
  Option: POSIX_CLOCKS_LIB (default = -lrt).

- SUNDIALS (optional), used when MFEM_USE_SUNDIALS = YES.
  Beginning with MFEM v3.3, SUNDIALS v2.7.0 is supported.
  Beginning with MFEM v3.3.2, SUNDIALS v3.0.0 is also supported.
  Beginning with MFEM v4.1, only SUNDIALS v5.0.0+ is supported.
  When MFEM_USE_CUDA is enabled, only SUNDIALS v5.4.0+ is supported.
  If MFEM_USE_MPI is enabled, we expect that SUNDIALS is built with support for
  both MPI and hypre.
  If MFEM_USE_CUDA is enabled, we expect that SUNDIALS is built with support
  for CUDA.
  URL: http://computation.llnl.gov/projects/sundials/sundials-software
  Options: SUNDIALS_OPT, SUNDIALS_LIB.
  Versions: SUNDIALS >= 5.0.0, SUNDIALS >= 5.4.0 for CUDA support.

- Mesquite (optional), used when MFEM_USE_MESQUITE = YES.
  URL: http://trilinos.org/oldsite/packages/mesquite
  Options: MESQUITE_OPT, MESQUITE_LIB.
  The Mesquite support is deprecated and will be removed in the future.

- SuiteSparse (optional), used when MFEM_USE_SUITESPARSE = YES.
  URL: http://faculty.cse.tamu.edu/davis/suitesparse.html
  Options: SUITESPARSE_OPT, SUITESPARSE_LIB.
  Versions: SuiteSparse >= 4.5.4, older versions may work too.

- SuperLU_DIST (optional), used when MFEM_USE_SUPERLU = YES. Note that
  SuperLU_DIST requires ParMETIS, which includes METIS 5 in its distribution.
  Both ParMETIS and the included METIS 5 should be built and installed in the
  same location. If using SuperLU_Dist v5, set MFEM_USE_SUPERLU5=YES.
  URL: http://crd-legacy.lbl.gov/~xiaoye/SuperLU
  Options: SUPERLU_OPT, SUPERLU_LIB.
  Versions: SuperLU_DIST >= 5.1.0.

- MUMPS (optional), used when MFEM_USE_MUMPS = YES. Note that MUMPS
  requires LAPACK, SCALAPACK and a reordering package such as PORD or METIS.
  URL: http://mumps.enseeiht.fr
  Options: MUMPS_OPT, MUMPS_LIB.
  Versions: MUMPS >= 5.1.1

- STRUMPACK (optional), used when MFEM_USE_STRUMPACK = YES. Note that STRUMPACK
  requires the PT-Scotch and Scalapack libraries as well as ParMETIS, which
  includes METIS 5 in its distribution. Starting with STRUMPACK v2.2.0, ParMETIS
  and PT-Scotch are optional dependencies.
  The support for STRUMPACK was added in MFEM v3.3.2 and it requires STRUMPACK
  2.0.0 or later.
  URL: http://portal.nersc.gov/project/sparse/strumpack
  Options: STRUMPACK_OPT, STRUMPACK_LIB.
  Versions: STRUMPACK >= 3.0.0.

- Ginkgo (optional), used when MFEM_USE_GINKGO = YES. Note that Ginkgo needs a
  C++ compiler that supports the C++-11 standard. For additional requirements
  and dependencies of specific modules see the Ginkgo webpage below.
  URL: https://ginkgo-project.github.io
  Options: GINKGO_OPT (Not used), GINKGO_LIB.

- AmgX (optional), used when MFEM_USE_AMGX = YES.
  URL: https://github.com/NVIDIA/AMGX
  Options: AMGX_OPT, AMGX_LIB.
  Versions: AmgX >= 2.1, older versions may work too.

- GnuTLS (optional), used when MFEM_USE_GNUTLS = YES. On most Linux systems,
  GnuTLS is available as a development package, e.g. gnutls-devel. On Mac OS X,
  one can get the library through the Homebrew package manager (http://brew.sh).
  URL: http://gnutls.org
  Options: GNUTLS_OPT, GNUTLS_LIB.
  Versions: GnuTLS >= 2.12.0, older versions may work too.

- NetCDF (optional), used when MFEM_USE_NETCDF = YES, required for reading Cubit
  mesh files. Also requires installation of HDF5 and ZLIB, as explained at the
  NetCDF web site. Note that we use the plain vanilla "C" version of NetCDF, you
  don't need the C++ or parallel versions.
  URL: www.unidata.ucar.edu/software/netcdf
  Options: NETCDF_OPT, NETCDF_LIB.
  Versions: NetCDF >= 4.4.0.

- PETSc (optional), used when MFEM_USE_PETSC = YES. Version 3.8 or higher of
  the PETSC dev branch is required. The MFEM and PETSc builds can share common
  libraries, e.g., hypre and SUNDIALS. Here's an example configuration, assuming
  PETSc has been cloned on the same level as mfem and hypre:
    ./configure --download-fblaslapack=yes --download-scalapack=yes \
                --download-mumps=yes --download-suitesparse=yes \
                --with-hypre-dir=../hypre-2.10.0b/src/hypre \
                --with-shared-libraries=0
  URL: https://www.mcs.anl.gov/petsc
  Options: PETSC_OPT, PETSC_LIB.
  Versions: PETSc >= 3.8.0.

- SLEPc (optional), used when MFEM_USE_SLEPC = YES. SLEPc depends on PETSc and
  uses some of the PETSc options when compiled.
  URL: https://slepc.upv.es/
  Options: SLEPC_OPT, SLEPC_LIB.
  Versions: SLEPc >= 3.8.0.

- Sidre (optional), part of LLNL's axom project, used when MFEM_USE_SIDRE = YES.
  Starting with MFEM v4.1, Axom version 0.3.1 or later is required.
  URL: https://github.com/LLNL/axom
       https://github.com/LLNL/conduit (Conduit)
       https://support.hdfgroup.org/HDF5 (HDF5)
  Options: SIDRE_OPT, SIDRE_LIB.
  Versions: Axom >= 0.3.1.

- Conduit (optional), used when MFEM_USE_CONDUIT = YES. Conduit Mesh Blueprint
  support requires Conduit >= v0.3.1 and VisIt >= v2.13.1 to read the output.
  URL: https://github.com/LLNL/conduit (Conduit)
       https://support.hdfgroup.org/HDF5 (HDF5)
  Options: CONDUIT_OPT, CONDUIT_LIB.
  Versions: Conduit >= 0.3.1.

- ADIOS2 (optional) used when MFEM_USE_ADIOS2 = YES.
  URL: https://adios2.readthedocs.io/
  Versions: ADIOS >= 2.5.0.

- PUMI (optional), used when MFEM_USE_PUMI = YES.
  URL: https://scorec.rpi.edu/pumi
       https://github.com/SCOREC/core
  Options: PUMI_OPT, PUMI_LIB.
  Versions: PUMI == 2.2.3.

- HiOp (optional), used when MFEM_USE_HIOP = YES.
  URL: https://github.com/LLNL/hiop
  Options: HIOP_OPT, HIOP_LIB.
  Versions: HIOP >= 0.1.

- GSLIB (optional), used when MFEM_USE_GSLIB = YES. The gslib library must be
  built prior to the MFEM build, as follows: download gslib-1.0.5, untar it at
  the same level as MFEM and create a symbolic link: "ln -s gslib-1.0.5 gslib".
  Build gslib in parallel or in serial based on the desired MFEM build: "make
  clean; make CC=mpicc" or "make clean; make CC=gcc MPI=0". Build MFEM with
  MFEM_USE_GSLIB=YES.
  URL: https://github.com/gslib/gslib/archive/v1.0.5.tar.gz
  Options: GSLIB_OPT, GSLIB_LIB.
  Versions: GSLIB >= 1.0.5.

- MKL CPardiso (optional), used when MFEM_USE_MKL_CPARDISO = YES.
  URL: https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html
  Options: MKL_CPARDISO_OPT, MKL_CPARDISO_LIB.
  Versions: Intel MKL >= 2020.

- CUDA (optional), used when MFEM_USE_CUDA = YES.
  URL: https://developer.nvidia.com/cuda-toolkit
  Options: CUDA_CXX, CUDA_ARCH, CUDA_OPT, CUDA_LIB.
  Versions: CUDA >= 9.1, older versions may work too.

- HIP (optional), used when MFEM_USE_HIP = YES.
  URL: https://rocm.github.io/ROCmInstall.html
  Options: HIP_CXX, HIP_ARCH, HIP_OPT, HIP_LIB.

- OCCA (optional), used when MFEM_USE_OCCA = YES.
  URL: https://libocca.org
  Options: OCCA_DIR, OCCA_OPT, OCCA_LIB.
  Versions: OCCA >= 1.1.0.

- libCEED (optional), used when MFEM_USE_CEED = YES.
  URL: https://github.com/CEED/libCEED
       https://ceed.exascaleproject.org/libceed
  Options: CEED_DIR, CEED_OPT, CEED_LIB.
  Versions: libCEED >= 0.7.

- RAJA (optional), used when MFEM_USE_RAJA = YES.
  Beginning with MFEM v4.3, only RAJA v0.13.0+ is supported.
  URL: https://github.com/LLNL/RAJA
  Options: RAJA_DIR, RAJA_OPT, RAJA_LIB.
  Versions: RAJA >= 0.13.0.

- Umpire, used when MFEM_USE_UMPIRE = YES.
  URL: https://github.com/LLNL/Umpire
  Options: UMPIRE_DIR, UMPIRE_OPT, UMPIRE_LIB.
  Versions: Umpire >= 2.0.0.

- MPFR (optional), used when MFEM_USE_MPFR = YES.
  URL: http://mpfr.org, it depends on the GMP library: https://gmplib.org
  Options: MPFR_OPT, MPFR_LIB.

- Libunwind (optional), used when MFEM_USE_LIBUNWIND = YES. The library is
  included with OS X (as of version 10.11). On Linux it could be installed with
  the libunwind-devel package.
  URL: http://www.nongnu.org/libunwind
  Options: LIBUNWIND_OPT, LIBUNWIND_LIB.

- ZLIB (optional), used when MFEM_USE_ZLIB = YES, or when MFEM_USE_NETCDF =
  YES (in the default settings for NETCDF_OPT and NETCDF_LIB).
  URL: https://zlib.net
  Options: ZLIB_OPT, ZLIB_LIB.

Building with CMake
===================
The MFEM build system consists of two steps: configuration and compilation.

The configuration step can be used to adjust paths to external libraries,
compilers, flags, etc, similar to any CMake build system. It is performed by
running

   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   cmake <mfem-source-dir> [OPTIONS] ...

The OPTIONS are of the form -D<VARIABLE>=<VALUE>, e.g. -DMFEM_USE_MPI=YES.
Detailed description of the configuration options is given below. Alternatively,
the options can be specified with an input file:

   cd <mfem-source-dir>/config
   cp defaults.cmake user.cmake
   (edit user.cmake)
   cd <mfem-build-dir>
   cmake <mfem-source-dir>

Note that user.cmake, if present, is loaded before defaults.cmake (and thus the
former takes precedence over the latter) and its path/name can be changed with

   cmake <mfem-source-dir> -DUSER_CONFIG=<user_config_file>

Debug and optimization options are controlled through the CMake variable
CMAKE_BUILD_TYPE which can be set to standard values like "Debug", and "Release"
(default).

To use a specific generator use the "-G <generator>" option of cmake:

   cmake <mfem-source-dir> -G "Xcode"
   cmake <mfem-source-dir> -G "Visual Studio 12 2013"
   cmake <mfem-source-dir> -G "MinGW Makefiles"

With CMake it is possible to build MFEM as a shared library using the standard
CMake option -DBUILD_SHARED_LIBS=1.

Once configured, the library can be built simply with (assuming a UNIX type
system, where the default is to generate "UNIX Makefiles")

   make -j 4
or
   cmake --build .
or
   cmake --build . --config Release  [Visual Studio, Xcode]

The build can be quick-tested by running

   make check
or
   cmake --build . --target check
or
   cmake --build . --config Release --target check  [Visual Studio, Xcode]

which will simply compile and run Example 1/1p. For more extensive tests that
check the results from all the serial/parallel MFEM examples and miniapps use:

   make exec -j 4
   make test
or
   cmake --build . --target exec
   cmake --build . --target test
or
   cmake --build . --config Release --target exec  [Visual Studio, Xcode]
   cmake --build . --config Release --target RUN_TESTS  [Visual Studio, Xcode]

Note that running all the tests may take a while.

Installation prefix can be configured by setting the standard CMake variable
CMAKE_INSTALL_PREFIX. To install the library, use

   make install
or
   cmake --build . --target install
or
   cmake --build . --config Release --target install  [Xcode]
   cmake --build . --config Release --target INSTALL  [Visual Studio]

The library will be installed in <PREFIX>/lib, the headers in <PREFIX>/include,
and the configuration CMake files in <PREFIX>/lib/cmake/mfem.


Configuration variables (CMake)
===============================
See the configuration file config/defaults.cmake for the default settings.

Note: the option MFEM_USE_CUDA requires CMake version 3.8 or newer!

Non-standard CMake variables for compilers:
   CXX    - If set, overwrite the auto-detected C++ compiler, serial build
   MPICXX - If set, overwrite the auto-detected MPI C++ compiler, parallel build

The compiler options for the various build types can be controlled using
standard CMake variables like CMAKE_CXX_FLAGS_RELEASE and CMAKE_CXX_FLAGS_DEBUG.

MFEM library features/options (CMake)
-------------------------------------
The following options are equivalent to the GNU make options with the same name:
[see "MFEM library features/options (GNU make)" above]

MFEM_USE_MPI
MFEM_USE_METIS - Set to ${MFEM_USE_MPI}, can be overwritten.
MFEM_USE_LIBUNWIND
MFEM_USE_LAPACK
MFEM_THREAD_SAFE
MFEM_USE_LEGACY_OPENMP
MFEM_USE_OPENMP
MFEM_USE_MEMALLOC
MFEM_TIMER_TYPE - Set automatically, can be overwritten.
MFEM_USE_MESQUITE
MFEM_USE_SUITESPARSE
MFEM_USE_SUPERLU
MFEM_USE_MUMPS
MFEM_USE_STRUMPACK
MFEM_USE_GINKGO
MFEM_USE_AMGX
MFEM_USE_GNUTLS
MFEM_USE_NETCDF
MFEM_USE_MPFR
MFEM_USE_ZLIB
MFEM_USE_PUMI
MFEM_USE_HIOP
MFEM_USE_CUDA
MFEM_USE_OCCA
MFEM_USE_CEED
MFEM_USE_RAJA
MFEM_USE_UMPIRE
MFEM_USE_SIDRE

The following options are CMake specific:

MFEM_ENABLE_TESTING  - Enable the ctest framework for testing.
MFEM_ENABLE_EXAMPLES - Build all of the examples by default.
MFEM_ENABLE_MINIAPPS - Build all of the miniapps by default.

External libraries (CMake):
---------------------------
For details about the external libraries, see the "External libraries (GNU
make)" section above.

The MFEM CMake build system provides auto-detection for some packages/libraries,
as listed below. The following configuration options are used/defined:
   <LIBNAME>_DIR
      Directory to search for <LIBNAME> first. The exact subdirectories
      searched, for headers and libraries, are chosen based on <LIBNAME>. If the
      library is not found in this location, then standard locations are
      searched.
   <LIBNAME>_REQUIRED_PACKAGES
      Specifies a list of package names that have to be explicitly added when
      linking with <LIBNAME> in addition to its main library, e.g. ParMETIS
      requires METIS, so we set ParMETIS_REQUIRED_PACKAGES to "METIS", see
      defaults.cmake.
   <LIBNAME>_INCLUDE_DIRS
      Location of the <LIBNAME> headers. Set by auto-detection, if successful.
      Can be set explicitly, e.g. if auto-detection fails.
   <LIBNAME>_LIBRARIES
      List of the <LIBNAME> library files/names/link-options. Set by
      auto-detection, if successful. Can be set explicitly, e.g. if
      auto-detection fails.

The CMake build system adds auto-detection for the following packages/libraries:

 - HYPRE
 - METIS - The option MFEM_USE_METIS_5 is auto-detected.
 - ParMETIS
 - MESQUITE
 - SuiteSparse
 - SuperLUDist, STRUMPACK
 - Ginkgo
 - AMGX
 - GNUTLS - Extends the built-in CMake support, to search GNUTLS_DIR as well.
 - NETCDF
 - MPFR
 - LIBUNWIND
 - POSIXCLOCKS
 - PUMI
 - HIOP
 - OCCA
 - RAJA
 - UMPIRE
 - AXOM - Used when MFEM_USE_SIDRE is enabled

The following built-in CMake packages are also used:

 - MPI, OpenMP, ZLIB
 - LAPACK, BLAS - Both are enabled via MFEM_USE_LAPACK. If auto-detection fails,
      set the <LIBNAME>_LIBRARIES option directly; the configuration option
      <LIBNAME>_DIR is not supported.


Building without GNU make or CMake
==================================
Before using another build system (e.g. Visual Studio) it is necessary to create
a proper configuration header file, config/config.hpp, using the template from
config/config.hpp.in:

   cp config/config.hpp.in config/_config.hpp

The file config/_config.hpp can then be edited to enable desired options.  The
MFEM library is simply a combination of all object files obtained by compiling
the .cpp source files in the source directories: general, linalg, mesh, and fem.


Specifying an MPI job launcher
==============================
By default, MFEM will use 'mpirun -np #' to launch any of its parallel tests or
miniapps, where # is the number of MPI tasks.  An alternate MPI launcher can be
provided by setting the MFEM_MPIEXEC and MFEM_MPIEXEC_NP config variables.

MFEM will expect the launcher command, plus the command line option to allow it
to specify a number of MPI tasks.

MFEM_MPIEXEC    = mpirun # default
MFEM_MPIEXEC_NP = -np    # default
MFEM_MPIEXEC    = srun   # example for platforms using SLURM
MFEM_MPIEXEC_NP = -n     # example for platforms using SLURM