Passing in Dynamically Sized User Data into Load Balancing

[jacobfen]

Hi All,

TL;DR: Is it possible to pass a user data object that contains dynamic data (E.g. a dynamic int array) into the load balancer?

ZIP of the all of the test files can be found here:

userdata_files.zip

Introduction

I am currently trying to pass dynamically sized user data to load balancing and am running into issues with what I presume to be with serialization of the data. I am using a modified test case found in: tests/charm++/load_balancing/lb_userdata_test

My LBUserDataStruct looks like the following:

class LBUserDataStruct {
  public:
  int idx;
  LDObjHandle handle;
  int size;
  int *arr;

  LBUserDataStruct() {}
  
  void pup(PUP::er &p) {
    p|idx;
    p|handle;
    p|size;

    if (p.isUnpacking()) {
      arr = new int[size];
    }

    PUParray(p, arr, size);
  }

  inline LBUserDataStruct &operator=(const LBUserDataStruct &indata) {
    size = indata.size;
    idx = indata.idx;
    handle = indata.handle;

    if (size > 0)
      arr = new int[size];
    
    for (int i = 0; i < size; i++)
      arr[i] = indata.arr[i];

    return *this;
  }

  ~LBUserDataStruct() {
    if (size > 0)
      delete [] arr;
  }
};

As you can see, I added a size integer (Called size) and an integer pointer (Called arr). Most of the code added is essentially a copy of the dynamic PUP examples outlined in the documentation (Found here: https://charm.readthedocs.io/en/latest/charm++/manual.html#dynamic-allocation).

Additionally, in Hello.C, I added the following lines to the same place as where the other user data was being set:

udata.size = 3;
udata.arr = new int[udata.size];
for (unsigned int i = 0; i < udata.size; i++) {
    udata.arr[i] = i + 1;
}

The work function for the TestUserDataLB load balancer is exactly the same as the normal load balancer except I added print statements for the size and arr variables:

CkPrintf("[%d] Array size: %d\n", i, udata->size);
CkPrintf("[%d] Array: ", i);
for (unsigned int j = 0; j < udata->size; j++) {
  CkPrintf("%d ", udata->arr[j]);
}
CkPrintf("\n");

The only other change made is the user data OBJ registration code in the constructor of the TestUserDataLB class:

CkpvAccess(_lb_obj_index) = LBRegisterObjUserData(sizeof(LBUserDataStruct) + (3 * sizeof(int)));

I am simply assuming that the dynamic integer array is three integers long (Since this object has to be registered with the correct size of the user data.

The Problem

When printing out the data in the work function of the load balancer, it only successfully reads data (From the arr array) from chare array elements that were on PE 0. The model will seg fault on any chare array elements that are on other PEs.

With the above changes, I run the following command: ./charmrun +p2 ++local ./hello 10 1000 +balancer TestUserDataLB

Error Output:

Charmrun> scalable start enabled. 
Charmrun> started all node programs in 0.036 seconds.
Charm++> Running in non-SMP mode: numPes 2
Converse/Charm++ Commit ID: bb387c6
Charm++> scheduler running in netpoll mode.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 unique compute nodes (8-way SMP).
Charm++> cpu topology info is gathered in 0.003 seconds.
[0] TestUserDataLB created
Running Hello on 2 processors for 10 elements for 1000 steps lb 10
[0] TestUserDataLB work
[0] tag 0 idx 0
[0] Array size: 3
[0] Array: 1 2 3 
[1] tag 1 idx 1
[1] Array size: 3
[1] Array: 1 2 3 
[2] tag 2 idx 2
[2] Array size: 3
[2] Array: 1 2 3 
[3] tag 3 idx 3
[3] Array size: 3
[3] Array: 1 2 3 
[4] tag 4 idx 4
[4] Array size: 3
[4] Array: 1 2 3 
[5] tag 5 idx 5
[5] Array size: 3
[5] Array: ------------- Processor 0 Exiting: Caught Signal ------------
Reason: segmentation violation

If you made it this far, thanks for reading. Any help is appreciated.

Jacob

[rbuch]

This is unsupported currently, the LB framework has no awareness of what your user data type is (as you can tell from LBRegisterObjUserData(sizeof(LBUserDataStruct) + (3 * sizeof(int))), it only allows a size to be passed in, not a type). The user data is stored as one big flat buffer with offsets that are provided at registration, which you can interpret as an object or struct by casting it a la LBUserDataStruct* udata = (LBUserDataStruct*)odata.getUserData(CkpvAccess(_lb_obj_index)) (_lb_obj_index is the provided offset).

Thus. the PUP routine that you write in your type is never executed (since the runtime doesn't know about your type); instead, it is PUP'd as a char array of the specified size. This obviously doesn't support pointers to data stored outside the user data buffer.

One thing you can do to approximate the ability to pass around dynamically sized data is to constrain the maximum length of your user data and allocate a static array of that max size in the object itself while using another variable to track how much of it is used.

[jacobfen]

Hi Ronak,

Thank you for the quick response. I suspected as much. I think your recommendation of a maximum size static array allocation is what I'll have to do then.

Thanks,

J