In-memory checkpoint with MPI

[jbakosi]

Hi folks,

I wonder if in-memory checkpointing works with the MPI layer.

I successfully hooked up CkStartMemCheckpoint() and my app runs correctly, displaying

CharmFT> Checkpointing...
CharmFT> Checkpoint finished in 0.036128 seconds.

but when I try to test it by killing one of the processes, MPI aborts the job with

--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

I tried setting a different OMPI error handler from a chare group constructor and also tried mpi_abort_delay via environment variables and the mpirun command line but none of these seem to have an effect.

Can this be done with MPI + Charm? Or should I try building my MPI-only libraries with AMPI?

Thanks,
Jozsef

[jbakosi]

I am guessing I can also try to demonstrate this with CkDieNow().

[jbakosi]

Okay, I have also tried failure injection with a kill-file, which correctly kills the PE specified in the file, but I get the same behavior as before, since MPI kills the whole program.

Hoping that adding spare processors my program would never get to the point of MPI abort, I have also tried adding spare processors using the +wp command line option, which apparently also works:

./charmrun +p4 ... +killFile rt.kill +wp 3
Charm++> FT using 3 processors and 1 spare processors.

But I guess since I'm running with MPI, this will also not work because underlying charmrun, mpirun is executed with -np 4, which gives me 4 MPI ranks, but then CkNumPes() in a group returns only 3, which causes a deadlock as, I guess, one of my MPI processes never returns from an MPI collective.

I also see in the manual that fault tolerance only works on the TCP-based net layers. I'm guessing this means it will not work with MPI, correct? If so, what options do I have for fault tolerance (other than getting rid of my MPI-libraries)?