Different outputs from main and print_matrix

[Goon83]

Hi All,
On the Mac (#673) with its default Makefile, I am testing the below simple example code to understand how DASH works. It gives me some confused results. I tried to print out matrix within main and print_matrix. sub function. The output from print_matrix looks wrong. Did I do something wrong with the code ?

Thanks.
Bin

🍀14:56:46@build🍃 mpirun -n 4  dash/ex.02.matrix.mpi
Matrix size: 8 x 8 == 64
Assigning matrix values
   0   1   2   3   4   5   6   7
   1   2   3   4   5   6   7   8
   2   3   4   5   6   7   8   9
   3   4   5   6   7   8   9   10
   4   5   6   7   8   9   10   11
   5   6   7   8   9   10   11   12
   6   7   8   9   10   11   12   13
   7   8   9   10   11   12   13   14
 rows = 8, cols = 8
Matrix:
     0     1     1     2     2     3     3     4
     4     5     5     6     6     7     7     8
     2     3     3     4     4     5     5     6
     6     7     7     8     8     9     9    10
     4     5     5     6     6     7     7     8
     8     9     9    10    10    11    11    12
     6     7     7     8     8     9     9    10
    10    11    11    12    12    13    13    14

//#include <unistd.h>
#include <iostream>
#include <cstddef>
#include <iomanip>

#include <libdash.h>
//#include <mpi.h>

using std::cout;
using std::endl;
using std::setw;

template<class MatrixT>
void print_matrix(const MatrixT & matrix) {
  typedef typename MatrixT::value_type value_t;
  auto rows = matrix.extent(0);
  auto cols = matrix.extent(1);

  cout << " rows = " << rows << ", cols = " << cols << std::endl;

  // Creating local copy for output to prevent interleaving with log
  // messages:
  value_t * matrix_copy = new value_t[matrix.size()];
  auto copy_end = std::copy(matrix.begin(),
                            matrix.end(),
                            matrix_copy);
  DASH_ASSERT(copy_end == matrix_copy + matrix.size());
  cout << "Matrix:" << endl;
  for (size_t r = 0; r < rows; ++r) {
    for (size_t c = 0; c < cols; ++c) {
      cout << " " << setw(5) << matrix_copy[r * cols + c];
      //cout << matrix[r][c] << " , ";
    }
    cout << endl;
  }
  delete[] matrix_copy;
}

int main(int argc, char* argv[])
{
  dash::init(&argc, &argv);

  size_t team_size    = dash::Team::All().size();
  dash::TeamSpec<2>  teamspec;
  teamspec.balance_extents();

  //int my_rank;
  //MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); // Find out process rank
 
  dash::global_unit_t myid   = dash::myid();
  size_t num_units   = dash::Team::All().size();
  //cout << "mpi_rank =" << my_rank << ", myid = " << myid << ", num_units  = " << num_units << std::endl;

  size_t tilesize_x  = 2;
  size_t tilesize_y  = 2;
  size_t rows = tilesize_x * num_units ;
  size_t cols = tilesize_y * num_units ;
  dash::Matrix<int, 2> matrix(
                         dash::SizeSpec<2>(
                           rows,
                           cols),
                         dash::DistributionSpec<2>(
                           dash::TILE(tilesize_x),
                           dash::TILE(tilesize_y)),
                         dash::Team::All(),
                         teamspec);
  size_t matrix_size = rows * cols;
  DASH_ASSERT(matrix_size == matrix.size());
  DASH_ASSERT(rows == matrix.extent(0));
  DASH_ASSERT(cols == matrix.extent(1));

  if (0 == myid) {
    cout << "Matrix size: " << rows
       << " x " << cols
       << " == " << matrix_size
       << endl;
  }

  // Fill matrix
  if (0 == myid) {
    cout << "Assigning matrix values" << endl;
    for(size_t i = 0; i < matrix.extent(0); i++) {
      for(size_t k = 0; k < matrix.extent(1); k++) {
        matrix[i][k] = i + k;
      }
    }
  }

  // Units waiting for value initialization
  dash::Team::All().barrier();

  // Read and assert values in matrix
  for (size_t i = 0; i < matrix.extent(0); ++i) {
    for (size_t k = 0; k < matrix.extent(1); ++k) {
      int value    = matrix[i][k];
      int expected = i+k;
      DASH_ASSERT(expected == value);
      if(myid == 1)
    	  cout << "   " << value ;
    }
    if(myid == 1)
    	cout << "\n";
  }

  dash::Team::All().barrier();

  // print matrix
  if (1 == myid) {
    print_matrix(matrix);
  }


  dash::finalize();
}

[devreal]

@fuchsto @rkowalewski @fuerlinger This is a question to you: what is the iteration order of a global iterator through a tiled matrix? My understanding is that it should still go row-wise, not tile-wise (the global pointer's iteration order is tile-wise, right?).

Iterating over the matrix using global iterators has the same result as what dash::copy puts into the local buffer (tile-wise iteration order):

template<class MatrixT>
void print_matrix_iter(const MatrixT& matrix)
{
  typedef typename MatrixT::value_type value_t;
  int c = 0;
  cout << std::endl;
  cout << "Matrix (iter):" << endl;
  for (auto it = matrix.begin(); it != matrix.end(); ++it) {
    std::cout << setw(5) << static_cast<value_t>(*it) << " ";
    if (++c % matrix.extent(0) == 0) {
      std::cout << std::endl;
    }   
  }
}

Matrix (iter):
    0     1     1     2     2     3     3     4 
    4     5     5     6     6     7     7     8 
    2     3     3     4     4     5     5     6 
    6     7     7     8     8     9     9    10 
    4     5     5     6     6     7     7     8 
    8     9     9    10    10    11    11    12 
    6     7     7     8     8     9     9    10 
   10    11    11    12    12    13    13    14 

[rkowalewski]

This is a question to you: what is the iteration order of a global iterator through a tiled matrix? My understanding is that it should still go row-wise, not tile-wise (the global pointer's iteration order is tile-wise, right?).

There is no single answer IMHO. It makes sense that we iterate tile-wise as the default iteration order, because...what should be the default iteration order of a matrix? There is no real standard how to iterate over a ND-Matrix which is why I would expect the user to explicitly ask for a global row-wise iterator.

Other maths projects (e.g. Eigen) also provide explicit iterators for such purposes 1. Unfortunately these concepts are not well supported in our library.

My conclusion is that tile-wise iteration is a valid default. The user has to explicitly iterate row-wise which is possible using the subscript operators at least. However, we could be better with more advanced iterators.

[devreal]

@rkowalewski Thanks for the clarification! I was wrongly assuming that global iterators follow the row-iteration order. We should definitely add this capability, maybe as a specialization of GlobIter? The use @Goon83 has is imo strong, copying rows out of a matrix is something people want to do at times and can be done much more efficient with dash::copy than element-wise in a raw for-loop.

@Goon83 I'm afraid at this point you're left to iterating over the in nested for loops and doing element-wise accesses. That is not super efficient but gets the job done. I hope we can come up with versions of global iterators that allow us to use dash::copy for this :)

[Goon83]

@devreal thanks for confirmation.

BTW, does the DASH has an API to access a contiguous subset of a matrix.
Say for a 2D matrix A with size 8 x 8, could I access the first 3 x 3 subset via A[0:2, 0:2]?

I did not find such API in https://codedocs.xyz/dash-project/dash/a01398.html

Thanks.