Suggestions for tracking down origins of "yaksa leaked handle pool objects"

[lindsayad]

Hi I was wondering if you had any suggestions for tracking down leaked handle pool objects. Normally I would do this kind of thing with valgrind, but the program I'm running is written in python and I do not trust the output. I track down PETSc object leaks using cli options like -log_view, but I believe I've fixed all those leaks, so now I'm at a loss for how to proceed.

[lindsayad]

Maybe @hzhou has some ideas? (Sorry just throwing some darts in the dark)

[hzhou]

The message is thrown by yaksa when there is a missing MPI_Type_free for a created datatype. Strictly speaking, this is a user issue since MPI requires all datatypes to be freed. Both MPI and yaksa cleans up the unfreed datatypes at Finalize so there is no memory leaks. We have modified yaksa to only issue this warning under debug build since this commit (#207) so you should not have this issue since mpich-4.1. Which version are you using?

[lindsayad]

(firedrake) lindad@pop-os:~/firedrake/src/FInAT(master)$ mpiexec --version
HYDRA build details:
    Version:                                 4.2.0
    Release Date:                            unreleased development copy
    CC:                              gcc      
    Configure options:                       '--disable-option-checking' '--prefix=/home/lindad/mpich/build-no-cuda/../installed-no-cuda' '--with-hwloc=embedded' '--without-cuda' '--cache-file=/dev/null' '--srcdir=../../../../src/pm/hydra' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -DNETMOD_INLINE=__netmod_inline_ofi__ -I/home/lindad/mpich/build-no-cuda/src/mpl/include -I/home/lindad/mpich/src/mpl/include -I/home/lindad/mpich/modules/json-c -I/home/lindad/mpich/build-no-cuda/modules/json-c -I/home/lindad/mpich/modules/hwloc/include -I/home/lindad/mpich/build-no-cuda/modules/hwloc/include -D_REENTRANT -I/home/lindad/mpich/build-no-cuda/src/mpi/romio/include -I/home/lindad/mpich/src/pmi/include -I/home/lindad/mpich/build-no-cuda/src/pmi/include -I/home/lindad/mpich/build-no-cuda/modules/libfabric/include -I/home/lindad/mpich/modules/libfabric/include -I/home/lindad/mpich/build-no-cuda/modules/yaksa/src/frontend/include -I/home/lindad/mpich/modules/yaksa/src/frontend/include'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Demux engines available:                 poll select

[lindsayad]

My mpich configure was pretty simple:

../configure --without-cuda --prefix=/home/lindad/mpich/build-no-cuda/../installed-no-cuda 

[hzhou]

To track down leaked datatype handle is a bit tough. I can add a special debug mode to dump some details of the leaked datatype handles. Let me know if this is a feature you'd like to have.

[lindsayad]

that sounds like it could be very useful!

[hzhou]

Great! I'll follow up here.