RBF mesh deformation: all-to-all communication step of boundary points for error calculation results in too large memory usage

[scheufks]

shared_exafsa/3dTube_MESH-L6_1 test case folder.

[davidsblom]

when doing weak scalability tests on supermuc, the error calculation step for the selection of points results in a all-to-all communication step of all points on the interface. this most probably results in a memory usage problem.

[davidsblom]

the solution would be to compute the errors locally, and communicate the locally selected point to the other cpus. the global selected point is then communicated to the other cpus.

[davidsblom]

Cause of problem:
With the current implementation of the RBF mesh deformation algorithm, all the boundary points are collected on each processor. This happens in the RBFMeshMotionSolver class during the initialisation step. Hence, the memory usage increases substantially per MPI rank for a weak scaling study.

Solution:
Complete rewrite of the RBFMeshMotionSolver, RBFCoarsening and RBFInterpolation class. This is probably not a small task. The meshes need to be distributed over the MPI ranks using Trilinos or PETSc.

[davidsblom]

My current idea is to use Elemental http://libelemental.org/ which implements distributed dense linear algebra algorithms. The API is similar to Eigen.

[davidsblom]

I've implemented a new ElRBFMeshMotion solver which uses the elemental library. All data is now distributed just like in PETSc. Hopefully, this case can now also be run on supermuc. Still updating the compilation scripts for supermuc.

[davidsblom]

@scheufks can you update the permissions of the files in /shared_exafsa/3dTube_MESH-L6_1? I cannot read all files.

[scheufks]

does it work now?

On 31.08.2016 14:53, David Blom wrote:

@scheufks https://github.com/scheufks can you update the permissions
of the files in |/shared_exafsa/3dTube_MESH-L6_1|? I cannot read all
files.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#258 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMH1xzG6r7IhllFGmRhP_Dxjl3J-2BaJks5qlXlPgaJpZM4INap-.

[davidsblom]

yes. thanks!

[davidsblom]

it runs now with the ElRBFMeshMotionSolver