added the reinflate api

[Anu-Ra-g]

Added the reinflate function to reinflate the index.

[martindurant]

There should be a one-line summary at the start of the docstring, since this is what help() and sphynx use as the title describing the method.

[martindurant]

The functions here are identical to the originals by @emfdavid ?

I am OK to include these now, but I think the next PR should move all of the tree/idx-related functions to another module (_grib_idx ?), and only import the public-facing functions into this module. As it is, the module is now extremely long compared to other formats, and it's not so obvious where a user should start.

[emfdavid]

Separate module and only exposing public functions sounds good to me.

[emfdavid]

Is there a better name for this function? I am terrible at names...

[martindurant]

Is there a better name for this function?

You mean "reinflate"? Maybe if you write a 2-sentence description, the name will present itself :)

[Anu-Ra-g]

@martindurant what should be the module name? The module should be included in a sub-package like kerchunk.grib.<module> or directly in the kerchunk.<module name>?

[martindurant]

I suggested _grib_idx above. Making kerchunk.grib into a package sounds like a bad idea.

[emfdavid]

Comments on naming (not my strong suit)... the Camus team had suggested "outbasing" as the name for this general approach of ingesting an index of chunks to a database and constructing the dataset/tree on the fly.

The "reinflate" method in particular builds the refspec dictionary for a zarr/xarray tree representing a set of grib data. The inputs are the static metadata for the refspec (everything but the chunks for variables with a time dimension), a set of chunk indexes for specific variables (usually stored in a DB), the aggregation type, and the set of axes that define how to arrange those chunks.

I wanted to be general, supporting all the FMRC slices which are represented in the AggregationType enum. The api allows passing a list of axes such that you can express a set of runtimes, a set of valid times, a set of horizons, or the 'best available' as of a certain time.

I hope there is some simpler expression of these concepts that might use this method as a backend, but I suspect you would have to make assumptions about the backend database/schema to wrap it up nicely.

[martindurant]

• This code should be in _grib_idx.py, as discussed
• There should be tests
• There needs to be far more documentation on how to use the code (again), else no one will ever touch these functions.

[emfdavid]

Thank you for getting the process started. I will plan to make progress this month, but it will be a nights and weekends effort. If that is not progressing by the end of October we can regroup.

[martindurant]

Sure, I was simply book-keeping :)

[rsignell]

@emfdavid I was trying to figure out where the 2024 GSoC work stands, and it looks like this is the blocker? Is that correct?

[martindurant]

Check out #523 - I think that's intended to finish off the topic.

[rsignell]

@martindurant ooh, yes indeed, that looks like it! Thanks!