Muemex

Note: These instructions are taken from the MueLu User's Guide, which should be viewed as the cannonical source of this information.

Muemex is MueLu's interface to the MATLAB environment. It allows access to a limited set of routines either MueLu as a preconditioner, Belos as a solver and Epetra or Tpetra for data structures. It is designed to provide access to MueLu's aggregation and solver routines from MATLAB and does little else. Muemex allows users to setup and solve arbitrarily many problems, so long as memory suffices. More than one problem can be set up simultaneously.

Cmake Configure and Make

To use Muemex, Trilinos must be configured with (at least) the following options:

export TRILINOS_HOME=/path/to/your/Trilinos/source/directory
cmake \
    -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON  \
    -D Trilinos_ENABLE_Amesos:BOOL=ON \
    -D Trilinos_ENABLE_Amesos2:BOOL=ON \
    -D Amesos2_ENABLE_KLU2:BOOL=ON \
    -D Trilinos_ENABLE_AztecOO:BOOL=ON \
    -D Trilinos_ENABLE_Epetra:BOOL=ON \
    -D Trilinos_ENABLE_EpetraExt:BOOL=ON \
    -D Trilinos_ENABLE_Fortran:BOOL=OFF \
    -D Trilinos_ENABLE_Ifpack:BOOL=ON \
    -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
    -D Trilinos_ENABLE_MueLu:BOOL=ON \
    -D Trilinos_ENABLE_Teuchos:BOOL=ON \
    -D Trilinos_ENABLE_Tpetra:BOOL=ON \
    -D TPL_ENABLE_MPI:BOOL=OFF \
    -D TPL_ENABLE_MATLAB:BOOL=ON \
    -D MATLAB_ROOT:STRING="<my matlab root>" \
    -D MATLAB_ARCH:STRING="<my matlab os string>" \
    -D Trilinos_EXTRA_LINK_FLAGS="-lrt -lm -lgfortran" \
${TRILINOS_HOME}

Since MueMex supports both the Epetra and Tpetra linear algebra libraries, you have to have both enabled in order to build MueMex. If you turn off either Epetra or Tpetra then you will run into an error message: MueMex requires Epetra, Tpetra and MATLAB.

Most additional options can be specified as well. It is important to note that Muemex does not work properly with MPI, hence MPI must be disabled in order to compile Muemex. The MATLAB_ARCH option is new to the cmake build system, and involves the MATLAB-specific architecture code for your system. There is currently no automatic way to extract this, so it must be user-specified. As of MATLAB 7.9 (R2009b), common arch codes are:

Code	OS
glnx86	32-bit Linux (intel/amd)
glnxa64	64-bit Linux (intel/amd)
maci64	64-bit MacOS
maci	32-bit MacOS

On 64-bit Intel/AMD architectures, Trilinos and all relevant TPLs (note: this includes BLAS and LAPACK) must be compiled with the -fPIC option. This necessitates adding:

-D CMAKE_CXX_FLAGS:STRING="-fPIC" \
-D CMAKE_C_FLAGS:STRING="-fPIC" \
-D CMAKE_Fortran_FLAGS:STRING="-fPIC" \

to the cmake configure line.

The additional linker flags specified in Trilinos_EXTRA_LINK_FLAGS may slightly vary depending on the system and the exact configuration. But the given parameters may work for most Linux based systems. If you encounter an error message like Target "muemex.mexa64" links to item "-Wl,-rpath-link,/opt/matlab/bin/glnxa64 " which has leading or trailing whitespace. You have to add some options to the Trilinos_EXTRA_LINK_FLAGS variable. At least adding -lm should be safe and fix the error message.

BLAS & LAPACK Option #1: Static Builds

Trilinos does not play nicely with MATLAB's default LAPACK and BLAS on 64-bit machines. If Muemex randomly crashes when you run with any Krylov method that has orthogonalization, chances are Muemex is finding the wrong BLAS/LAPACK libraries. This leaves you with one of two options. The first is to build them both statically and then specify them as follows:

-D LAPACK_LIBRARY_DIRS:STRING="<path to my lapack.a>" \
-D BLAS_LIBRARY_DIRS:STRING="<path to my blas.a>" \

Using static linking for LAPACK and BLAS prevents MATLAB's default libraries to take precedence.

##BLAS & LAPACK Option #2: LD_PRELOAD \label{sec:preload} The second option is to use LD_PRELOAD to tell MATLAB exactly which libraries to use. For this option, you can use the dynamic libraries installed on your system. Before starting MATLAB, set LD_PRELOAD to the paths of libstdc++.so corresponding to the version of GCC used to build Trilinos, and the paths of libblas.so and liblapack.so on your local system.

For example, if you use bash, you'd do something like this

export LD_PRELOAD=<path>/libstdc++.so:<path>/libblas.so:<path>/liblapack.so

For csh / tcsh, do this

setenv LD_PRELOAD <path>/libstdc++.so:<path>/libblas.so:<path>/liblapack.so

Running MATLAB

Before you run MATLAB you have to make sure that MATLAB is using the same libraries that have been used for compiling Muemex. This includes the libstdc++.so and depending whether you turned on/off fortran also libgfortran.so. Please make sure that the correct libraries and paths are declared in the LD_PRELOAD variable. You can refer to section \ref{sec:preload} to see how the LD_PRELOAD variable is set.

For a 64 bit Linux system using the bash the command should look like

export LD_PRELOAD=/usr/lib64/libstdc++.so.6:/usr/lib64/libgfortran.so.3:$LD_PRELOAD

to add the libstdc++.so and libgfortran.so to the existing LD_PRELOAD variable. Then run the MATLAB executable in the same shell window. Note, that this step is necessary even if you statically linked BLAS and LAPACK.

If you are unsure which libraries have to be set in the LD_PRELOAD variable you will find out latest if you start MATLAB and try to run Muemex. It will throw some error messages with the missing library names. For a 64 bit Linux system the standard libraries usually can be found in /usr/lib64 or /usr/lib (for a 32 bit system).

Using Muemex

Muemex is designed to be interfaced with via the MATLAB script muelu.m. There are five modes in which Muemex can be run:

1. Setup Mode --- Performs the problem setup for MueLu. Depending on whether or not the Linear Algebra_ option is used, Muemex creates either an unpreconditioned Epetra problem, an Epetra problem with MueLu, or a Tpetra problem with MueLu. The default is tpetra. The epetra mode only supports real-valued matrices, while tpetra supports both real and complex and will infer the scalar type from the matrix passed during setup. This call returns a problem handle used to reference the problem in the future, and (optionally) the operator complexity, if a preconditioner is being used.
2. Solve Mode --- Given a problem handle and a right-hand side, Muemex solves the problem specified. Setup mode must be called before solve mode.
3. Cleanup Mode --- Frees the memory allocated to internal MueLu, Epetra and Tpetra objects. This can be called with a particular problem handle, in which case it frees that problem, or without one, in which case all Muemex memory is freed.
4. Status Mode --- Prints out status information on problems which have been set up. Like cleanup, it can be called with or without a particular problem handle.
5. Get Mode --- Get information from a MueLu hierarchy that has been generated. Given the problem handle, a level number and the name of the field, returns the appropriate array or scalar as a MATLAB object.

All of these modes, with the exception of status and cleanup take option lists which will be directly converted into Teuchos::ParameterList objects by Muemex, as key-value pairs. Options passed during setup will apply to the MueLu preconditioner, and options passed during a solve will apply to Belos.

Setup Mode

Setup mode is called as follows:

>> [h, oc] = muelu('setup', A[, 'parameter', value,...])

The parameter A represents the sparse matrix to perform aggregation on and the parameter/value pairs represent standard MueLu options.

The routine returns a problem handle, h, and the operator complexity oc for the operator. In addition to the standard options, setup mode has one unique option of its own:

**Linear Algebra** [string] Whether to use 'epetra unprec', 'epetra', or 'tpetra'. Default is 'epetra' for real matrix and 'tpetra' for complex matrix.

Solve Mode

Solve mode is called as follows:

>> [x, its] = muelu(h[, A], b[, 'parameter', value,...])

The parameter h is a problem handle returned by the setup mode call, A is the sparse matrix with which to solve and b is the right-hand side. Parameter/value pairs to configure the Belos solver are listed as above. If A is not supplied, the matrix provided when setting up the problem will be used. x is the solution multivector with the same dimensions as b, and its is the number of iterations Belos needed to solve the problem.

All of these options are taken directly from Belos, so consult its manual for more information. Belos output style and verbosity settings are implemented as enums, but can be set as strings in Muemex. For example:

>> x = muelu(0, b, 'Verbosity', 'Warnings + IterationDetails','Output Style', 'Brief');

Verbosity settings can be separated by spaces, '+' or ','. Belos::Brief is the default output style.

Cleanup Mode

Cleanup mode is called as follows:

>> muelu('cleanup'[, h])

The parameter h is a problem handle returned by the setup mode call and is optional. If h is provided, that problem is cleaned up. If the option is not provided all currently set up problems are cleaned up.

Status Mode

Status mode is called as follows:

>> muelu('status'[, h])

The parameter h is a problem handle returned by the setup mode call and is optional. If h is provided, status information for that problem is printed. If the option is not provided all currently set up problems have status information printed.

Get Mode

Get mode is called as follows:

>> muelu('get', h, level, fieldName[, typeHint])

The parameter h is the problem handle, and level is an integer that identifies the level within the hierarchy containing the desired data. fieldName is a string that identifies the field within the level, e.g. 'Nullspace'. typeHint is an optional parameter that tells MueMex what data type to expect from the level. This is a string, with possible values 'matrix', 'multivector', 'lovector' (ordinal vector), or 'scalar'. MueMex will attempt to guess the type from fieldName but typeHint may be required.

Tips and Tricks

Internally, MATLAB represents all data as doubles unless you go through efforts to do otherwise. Muemex detects integer parameters by a relative error test, seeing if the relative difference between the value from MATLAB and the value of the int-typecast value are less than 1e-15. Unfortunately, this means that Muemex will choose the incorrect type for parameters which are doubles that happen to have an integer value (a good example of where this might happen would be the parameter `smoother Chebyshev: alpha', which defaults to 30.0). Since Muemex does no internal typechecking of parameters (it uses MueLu's internal checks), it has no way of detecting this conflict. From the user's perspective, avoiding this is as simple as adding a small perturbation (greater than a relative 1e-15) to the parameter that makes it non-integer valued.