Link SECM MOM6 workflow as a case study?

[martindurant]

https://ncar.github.io/esds/posts/2023/kerchunk-mom6/
@dcherian

[dcherian]

👍 it is on my very long list

[dcherian]

FYI I didn't dig too deep but these are netCDF3 files and there seemed like a slowdown on actual data reads during a workflow. So perhaps #123 (comment)?

[martindurant]

I think what we really need is partial reads in zarr, which should be simple for uncompressed data like this. A thing promised but not yet delivered.

[dcherian]

I think what we really need is partial reads in zarr, which should be simple for uncompressed data like this. A thing promised but not yet delivered.

how would I confirm this? Would it be better if I used "subchunking"?

[martindurant]

Yes, you could certainly use subchunking, which would get you better random access at the cost of (many) more references.

how would I confirm this?

Turn on logger, perhaps like

fsspec.util.setup_logging(logger_name="fsspec.reference")