Global interpolation matrix --> MPI-distributed interpolation matrices #258

sbrdar · 2025-02-13T17:34:00Z

add method 'distribute_global_matrix (src_fs, tgt_fs, global_matrix, mpi_root)' to return local interpolation matrices
make sure that interpolation method in Atlas which take cache matrix as an argument actually use that cache
extend test_interpolation_global_matrix to: 1) create a global matrix from a parallel interpolator (was there before); 2) distribute the global matrix to task-local matrices (new); 3) compare the remapped field gathered in a global field on the single task with the global-matrix remapped global field.
extend sandbox/atlas-global-matrix to be able to read from the disc and distribute matrix to the parallel interpolator

codecov-commenter · 2025-02-14T00:07:21Z

Codecov Report

Attention: Patch coverage is 78.22581% with 81 lines in your changes missing coverage. Please review.

Project coverage is 80.39%. Comparing base (e351a31) to head (3b571d9).
Report is 54 commits behind head on develop.

Files with missing lines	Patch %	Lines
...tured/ConservativeSphericalPolygonInterpolation.cc	45.71%	19 Missing ⚠️
...on/method/structured/StructuredInterpolation3D.tcc	0.00%	13 Missing ⚠️
...rc/atlas/interpolation/method/knn/GridBoxMethod.cc	0.00%	11 Missing ⚠️
src/atlas/linalg/sparse/SparseMatrixStorage.cc	69.23%	8 Missing ⚠️
.../interpolation/test_interpolation_global_matrix.cc	94.44%	6 Missing ⚠️
...atlas/interpolation/method/knn/NearestNeighbour.cc	60.00%	4 Missing ⚠️
...on/method/structured/StructuredInterpolation2D.tcc	69.23%	4 Missing ⚠️
...interpolation/method/unstructured/FiniteElement.cc	66.66%	4 Missing ⚠️
.../method/unstructured/UnstructuredBilinearLonLat.cc	69.23%	4 Missing ⚠️
src/atlas/interpolation/method/binning/Binning.cc	0.00%	2 Missing ⚠️
... and 3 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #258      +/-   ##
===========================================
+ Coverage    79.82%   80.39%   +0.57%     
===========================================
  Files          815      850      +35     
  Lines        64522    68917    +4395     
===========================================
+ Hits         51502    55404    +3902     
- Misses       13020    13513     +493

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…o parallel Interpolation

…dle with creation of FunctionSpace from the conservative-spherical-polygon interpolator (not done)

…terpolation

…re that the interpolation setup from cache does use that cache (with Willem D.)

…sts for interpolations which do not accept a cached matrix

… the global matrix from the disc

…nd ./bin/atlas-global-matrix --read

… modus

…al-matrix tools

… ATLAS_TRACEs

wdeconinck · 2025-02-18T10:39:18Z

Thanks @sbrdar , I have rebased this on latest develop, which now contains a much tougher ci that no longer permits warnings. Please fix the warnings before I review.

wdeconinck

Very nice work!
I have a few suggestions, see below.

src/atlas/interpolation/AssembleGlobalMatrix.cc

src/atlas/interpolation/method/cubedsphere/CubedSphereBilinear.h

wdeconinck · 2025-02-20T16:34:14Z

src/atlas/interpolation/method/knn/KNearestNeighbours.cc

+        setMatrix(cache);
+        source_ = source;
+        target_ = target;
+        buildPointSearchTree(source);


Building the PointSearchTree should not be needed when the cache contains a matrix.

On the other hand when there's no matrix in the cache this sets up nothing. That needs fixing.

if(interpolation::MatrixCache(cache)) { ... } else { THIS SITUATION }

src/atlas/interpolation/method/knn/NearestNeighbour.cc

wdeconinck · 2025-02-20T16:37:36Z

src/atlas/interpolation/method/structured/StructuredInterpolation2D.tcc

+        do_setup(source, target);
+        return;
+    }
+    ATLAS_NOTIMPLEMENTED;


This needs also the handling of the "matrix_free" configuration option, meaning that when "matrix_free" is chosen we also need to ignore the matrix.

This if is not needed, and you can just call do_setup(source, target)

Thanks! Now, if the method is not matrix free and there is a cache matrix, we initialise from the cache. In all other cases do_setup(source, target)

src/atlas/interpolation/method/structured/StructuredInterpolation3D.tcc

wdeconinck · 2025-02-20T16:39:58Z

src/atlas/interpolation/method/unstructured/FiniteElement.cc

+        ATLAS_ASSERT(matrix().cols() == source.size());
+        return;
+    }
+    if (functionspace::NodeColumns(source) && functionspace::PointCloud(target)) {


This if is not needed, and you can just call do_setup(source, target)

Thanks! Done.

src/atlas/interpolation/method/unstructured/UnstructuredBilinearLonLat.cc

sbrdar requested a review from wdeconinck February 13, 2025 17:34

sbrdar and others added 23 commits February 18, 2025 11:26

test setup to code: from global matrix with the two function spaces t…

3a421d8

…o parallel Interpolation

prepare assemble distribute for more cases

8d46f76

setup the distribution of the global matrix - to be finalised still

e38190e

allow make_view on zero size matrix

02406c4

implement distribute_global_matrix up to halo

963b749

add Intepolation(Config, FunctionSpace, FunctionSpace, Cache) and fid…

4063277

…dle with creation of FunctionSpace from the conservative-spherical-polygon interpolator (not done)

fix the unit test to compare the global matrix and the distributed in…

87fb2de

…terpolation

1) fix more bugs in the distribution of the global matrix; 2) make su…

a2579e1

…re that the interpolation setup from cache does use that cache (with Willem D.)

improve unit test for distribution of global matrices; comment out te…

bf47bf3

…sts for interpolations which do not accept a cached matrix

comment out the intermediate check

261e6da

fix usage of Cache in different interpolation methods

9dc524b

keep the unit test under 5 sec exection time

a40953e

clean up

83f0dde

make sandbox tool -atlas-global-matrix- work parallel when reading in…

ef6ddbd

… the global matrix from the disc

cleanup and but fix in the output from the interpolation

44b5f4b

make default naming work, i.e. Both work: ./bin/atlas-global-matrix a…

9577c62

…nd ./bin/atlas-global-matrix --read

cleanup

14a4658

add timers in sandbox tool atlas-global-matrix when in the reading-in…

07838c7

… modus

cleanup in atlas-global-matrix

dda030d

another cleanup in atlas-global-matrix

ccdb644

fix interpolation setup from cache for the nearest-neighbour class

5bd8055

make the knn methods and grid-box-method work again in the atlas-glob…

5a058ec

…al-matrix tools

1) use unordered_map in the distribution of the global matrix; 2) add…

efc5b6c

… ATLAS_TRACEs

wdeconinck force-pushed the feature/serialized-sparsematrix branch from 1b786cd to efc5b6c Compare February 18, 2025 10:26

remove warnings

4cb4448

wdeconinck requested changes Feb 20, 2025

View reviewed changes

fix for the review #256

b673b30

wdeconinck approved these changes Feb 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Global interpolation matrix --> MPI-distributed interpolation matrices #258

Global interpolation matrix --> MPI-distributed interpolation matrices #258

sbrdar commented Feb 13, 2025

codecov-commenter commented Feb 14, 2025 •

edited

Loading

wdeconinck commented Feb 18, 2025 •

edited

Loading

wdeconinck left a comment

wdeconinck Feb 20, 2025

wdeconinck Feb 20, 2025

sbrdar Feb 20, 2025

wdeconinck Feb 20, 2025

sbrdar Feb 20, 2025

Global interpolation matrix --> MPI-distributed interpolation matrices #258

Are you sure you want to change the base?

Global interpolation matrix --> MPI-distributed interpolation matrices #258

Conversation

sbrdar commented Feb 13, 2025

codecov-commenter commented Feb 14, 2025 • edited Loading

Codecov Report

wdeconinck commented Feb 18, 2025 • edited Loading

wdeconinck left a comment

Choose a reason for hiding this comment

wdeconinck Feb 20, 2025

Choose a reason for hiding this comment

wdeconinck Feb 20, 2025

Choose a reason for hiding this comment

sbrdar Feb 20, 2025

Choose a reason for hiding this comment

wdeconinck Feb 20, 2025

Choose a reason for hiding this comment

sbrdar Feb 20, 2025

Choose a reason for hiding this comment

codecov-commenter commented Feb 14, 2025 •

edited

Loading

wdeconinck commented Feb 18, 2025 •

edited

Loading