Return just data when loading .topostats

[ns-rse]

Described in the pull to add a grains entry point the structure of a topostats object during processing with topostats is different from that which is returned by AFMReader.topostats.load_topostats().

At the end of processing a topostats object is a nested dictionary of (mostly) Numpy arrays which is written to HDF5 as is.

However, on loading with AFMReader.topostats.load_topostats() this dictionary is loaded and two items extracted from the data dictionary (which is the structure as written to HDF5) and a tuple returned.

This can be seen from line 42 onwards where the flattened image is extracted along with pixel_to_nm_scaling and returned along with the complete data as a tuple.

    try:
        with h5py.File(file_path, "r") as f:
            data = unpack_hdf5(open_hdf5_file=f, group_path="/")
            if data["topostats_file_version"] >= 0.2:
                data["img_path"] = Path(data["img_path"])
            file_version = data["topostats_file_version"]
            logger.info(f"[{filename}] TopoStats file version : {file_version}")
            image = data["image"]
            pixel_to_nm_scaling = data["pixel_to_nm_scaling"]

    except OSError as e:
        if "Unable to open file" in str(e):
            logger.error(f"[{filename}] File not found : {file_path}")
        raise e

    return (image, pixel_to_nm_scaling, data)

It would make more sense if a topostats object were the same structure, whether it is created during processing or loaded using AFMReader. I believe the current situation is for "convenience" but extraction of either parameter if only the data were returned would not be wholly different, i.e. referring to the required element of a dictionary by name.

Suggest removing this and having instead...

    try:
        with h5py.File(file_path, "r") as f:
            data = unpack_hdf5(open_hdf5_file=f, group_path="/")
            if data["topostats_file_version"] >= 0.2:
                data["img_path"] = Path(data["img_path"])
            file_version = data["topostats_file_version"]
            logger.info(f"[{filename}] TopoStats file version : {file_version}")

    except OSError as e:
        if "Unable to open file" in str(e):
            logger.error(f"[{filename}] File not found : {file_path}")
        raise e

    return data

[SylviaWhittle]

I'm tentatively in favour of the proposed change there, I am trying to think of any issues but can't come up with any, though it seems like the kind of thing that might bring up unforeseen complications 😅

Adding all the data that is produced during processing would be good.

Do you think I should add the image mask tensor to the .topostats file in my PR or leave it for its own PR?

[ns-rse]

There will always be something we can't anticipate!

I'd leave the image mask tensor as a separate PR, grains refactor is a behemoth already!