Unusual fi object behavior (V3.1.1)

[mr483]

I am working to parallelize some fi.calculate_wake() calls in FLORIS V3.1.1, and have generally the structure shown below (this is my attempt at a minimum reproducible example, to show the principle of the problem).

Effectively, I am configuring a bunch of floris instances with different sets of parameters (using copy.deepcopy() to create unique objects with those settings) and then submitting them to the starmap function. "s" is a label, which makes more sense in full context of the use-case. Now, if I were to run fi.calculate_wake() on the floris instances directly after they are configured (i.e., after fi.reinitialize(...)), everything runs fine. However, within the ts_get_sub_AEPs() function, I get the error:

AttributeError: 'Farm' object has no attribute 'turbine_type_names_sorted'

Within the parallelized function, I added some debugging code and determine the fi instances have the correct wake model parameters and tuning parameters which I have configured. However, they are missing some critical attributes (such as turbine_type_names_sorted). I had the function just return the object floris_inst directly and examined the results, and indeed the input fi which is provided to multiargs does not match the fi which exists in the ts_get_sub_AEPs function, despite there being no intervening operations done with the fi object.

I'm probably missing something very fundamental here, but not sure what it is. The quick fix is that within the function that is parallelized, I just add a fi.reinitialize(), with no change of parameters. It then correctly creates the sorted arrays which I need. But, this of course adds a little overhead, and I can't understand why it's necessary since I haven't done any operations on the fi object which would seemingly necessitate me to reinitialize.

import mpi4py.futures as mp

def ts_get_sub_AEPs(s, floris_inst):
    floris_inst.calculate_wake()
    pow = floris_inst.get_turbine_powers()

    return pow


if __name__ == "__main__":
  
  fi = FlorisInterface(config_file)
          # add layout, coming from separate dataframe
          fi.reinitialize(layout=(layout['layout_x'], layout['layout_y']),
                                  turbine_type=layout['turbine_type'])
  
          multiargs = []
          for s in set:
              fi = copy.deepcopy(fi)
              fi = configure_FL_tuning(fi, tunings, s, model)
              for i in range(conditions):
                  fi = copy.deepcopy(fi)
                  fi.reinitialize(wind_directions=... ,)
                  multiargs.append((s,fi))
  
  _PoolExecutor = mp.MPIPoolExecutor
  
  with _PoolExecutor(max_workers=len(multiargs)) as p:
      out = p.starmap(ts_get_sub_AEPs, multiargs)

[mr483]

After some further thought + debugging, I think it is likely something along the lines of this answer (https://stackoverflow.com/questions/46813375/multiprocessing-pool-map-drops-attribute-of-subclassed-ndarray), which is that the extra attributes are lost during serialization.

[misi9170]

Hi @mr483,

I'm no multiprocessing expert, but yes, it does seem likely to be something to do with the serialization. I assume that the missing attribute turbine_type_names_sorted is actually on the farm attribute of floris (i.e., several layers down into the code architecture, fi.floris.farm.turbine_type_names_sorted).

I am a bit perplexed at first about why the turbine_type_names_sorted attribute would appear as missing, when other attributes immediately beforehand presumably are found as they do not raise an error (e.g. pPs_sorted)---however, I notice that the turbine_type_names_sorted are not declared in the list of attributes at the beginning of the Farm class, whereas most others are; likely because they are not actually used outside of the Farm class, it seems. If I'm right about that, you could try a couple of small changes to the Farm class that may fix this particular problem:

1. Add turbine_type_names_sorted: NDArrayFloat = field(init=False, default=[]) to the list of attributes declared at the top of the Farm class.
   OR
2. Change the lines

        self.turbine_type_names_sorted = [turb["turbine_type"] for turb in self.turbine_definitions]
        self.turbine_type_map_sorted = np.take_along_axis(
            np.reshape(self.turbine_type_names_sorted * n_wind_directions, np.shape(sorted_coord_indices)),
            sorted_coord_indices,
            axis=2
        )

to

        turbine_type_names_sorted = [turb["turbine_type"] for turb in self.turbine_definitions]
        self.turbine_type_map_sorted = np.take_along_axis(
            np.reshape(turbine_type_names_sorted * n_wind_directions, np.shape(sorted_coord_indices)),
            sorted_coord_indices,
            axis=2
        )

that is, make turbine_type_names_sorted simply a variable rather than an attribute of Farm.

Those might help solve the immediate problem, but you could run into serialization issues elsewhere. To be honest, I think the best approach is probably to simply call fi.reinitialize() within your parallelized function, as you've already suggested (as you said, there is possibly some extra overhead here, but it's likely it'll be small compared to the calculate_wake() call if you're passing many conditions to evaluate).

[mr483]

Hi @misi9170 , thanks a lot for your thoughts and suggestion. I gave it a shot, but indeed this just results in the same error for the next missing attribute, 'turbine_power_interps'. Anyway, as you say, the reinitialize is relatively fast compared to the wake calculation, so I'll stick with that solution for now. Thanks again.