Write to Zarr

[srivarra]

Description of new feature

Hello, thanks for making awkward! I'd love a way to save an awkward Array to the Zarr format if possible! I've checked the links and searched for Zarr and found a discussion post

This would be specifically beneficial for Image Mass Spectrometry, where we may have tens of thousands to hundreds of thousands of pixels, each having variable length masses and intensities (this is where awkward has been helping me).

If this is a feature worth considering / in the scope of awkward, I'd also love to help with a contribution or two!

[jpivarski]

A single (non-rectangular) Awkward Array could be encoded in Zarr by breaking it down into buffers, saving those as one-dimensional arrays in Zarr, and then reconstituting it from Zarr. Earlier discussions with the Zarr team were about defining a standard way to do that, so that the Zarr object would declare itself to be an Awkward Array, because the intermediate buffers are not meaningful by themselves.

This documentation describes the process of breaking an Awkward Array into buffers and reconstituting it, with an example of saving it to HDF5. HDF5 has the same problem—if someone opens the HDF5 file of raw buffers without knowing that it's supposed to be interpreted as an Awkward Array, they wouldn't necessarily be able to make sense of its contents. (That's why we don't have a standard way to save to HDF5.)

I haven't been following Zarr developments recently. Is it looking like a version 3 extension could mark a collection of differently sized, differently partitioned buffers as a higher-level object that should be interpreted by the Awkward library?

Alternatively, do you need this for your own purposes, such that the encoding does not need to be standardized? (That could be done by just following the above-linked documentation, replacing Zarr for HDF5, and wouldn't need anything to be contributed centrally.)

[srivarra]

Yeah I just needed this for my own purposes, so it didn't need to be standardized. I'll go ahead give your suggestion a shot, thank you!