New Feature: date display mapping

[SpacemanPaul]

Description

The overlapping-time-period support in #933 works by using (midnight on) the start date for search and display purposes.

E.g. 2019-01-01T00:00:00.000Z to 2021-12-31T23:59:59.998Z in the ODC index:

• exposed as 2019-01-01 in WMS clients

And when a WMS request comes in with 2019-01-01, the ODC search is:

• start time is 2019-01-01

Users would like to be able to display, for example, the MIDDLE year of a 3-year rolling summary in WMS (2020-01-01 in the above example.)

Proposal

At minimum we would need something like:

    "time_resolution": "summary",
    # The new fields below are only allowed with "summary" time resolution.
    "time_odc_to_ows" : {
         # This is a user supplied function that takes a datetime.datetime object which is the start date
         # and returns a datetime.date object to use in web service requests and responses.
         # Default is equivalent to:
         #      lambda start: start.date()
         "function": "myconfig.functions.add_one_year",
    },
    "time_ows_to_odc": {
         # This is a user supplied function that takes a datetime.date object as used in web service requests and responses
         # and returns a datetime.datetime object which is to be compared to the start_date in ODC queries.
         # Default is equivalent to:
         #      lambda start: datetime.datetime(start.year, start.month, start.day, tzinfo=utc)
         "function": "myconfig.functions.subtract_one_year",
    }

NOTE: time_odc_to_ows and time_ows_to_odc MUST be inverses of each other. Update ranges will fail if this is not true for the start times of any datasets in your index. For this reason you must supply both or neither.

[mickwelli]

Thanks @SpacemanPaul, I think this addresses the issue. It is notable that the middle date is returned as the time variable when users query odc, e.g. in this notebook the first image has a time label of 2020-12-16T23:59:59.. My view is that images in overlapping products (esp. 3-monthly) should be referred to by their mid-point for consistency and to avoid confusion.

https://github.com/digitalearthafrica/deafrica-sandbox-notebooks/blob/main/Datasets/Rolling_GeoMAD.ipynb

[SpacemanPaul]

I would strongly disagree - I think using the exact middle point confronts the user with unnecessary complexity and is confusing (esp. for 3-monthly). First of all you do not want to presenting irregular time parts (23:59:59 or worse) for monthly data in WMS - that would be ugly, cluttered and confusing. Secondly, as month-lengths vary, the middle date will vary from around the 14th to around the 16th from period to period. This variation makes the data look less uniform in reporting than it is, another potential source of confusion.

For overlapping 3-month monthly windows, I think the start date or the start of the middle month make the most sense - but the implementation described in this issue would allow centre-dates if you are dead set on them. But time parts in the OWS datestamp are still only allowed for layers with more than one time-slice per day.

It is always safest to assume that most people accessing data through WMS are not familiar with working with the ODC directly.

[mickwelli]

Thanks @SpacemanPaul, that makes sense. Start of middle month sounds good. Our science team have noticed folks referring to images in this product by the middle month most often e.g. 'June rolling geomedian image' to refer to May-Jul.

I think mid-point requests have come because images display incorrectly for some of our users due to a TerriaJS timezone issue TerriaJS/terriajs#6822. That is, the start of month date shows as end of previous month for users in UTC- territory, but I realise that is a separate issue.

[robbibt]

I would strongly disagree - I think using the exact middle point confronts the user with unnecessary complexity and is confusing (esp. for 3-monthly). First of all you do not want to presenting irregular time parts (23:59:59 or worse) for monthly data in WMS - that would be ugly, cluttered and confusing. Secondly, as month-lengths vary, the middle date will vary from around the 14th to around the 16th from period to period. This variation makes the data look less uniform in reporting than it is, another potential source of confusion.

For overlapping 3-month monthly windows, I think the start date or the start of the middle month make the most sense - but the implementation described in this issue would allow centre-dates if you are dead set on them. But time parts in the OWS datestamp are still only allowed for layers with more than one time-slice per day.

For our DEA Intertidal rolling 3-year use case, the start of the middle year would work well. I agree that using the precise mid-point would add more noisy clutter than necessary! (although with this proposal, it looks like a user could easily provide a "add 1.5 years" lambda func as well if they wanted to).

One of the reasons we need to customise the displayed date is that our products capture environments that change rapidly over the three-year epoch. Because we use medians, they strongly suppress the start and end of the timeseries, and effectively represent conditions much closer to the mid-point of the epoch (e.g. 2020 for a 2019-2021 epoch). Because of that, the start date isn't an appropriate choice and is likely to mislead users - the data is much more representative of the middle date.

(I think the same logic would apply to DE Africa's rolling GeoMAD - in rapidly changing areas, those layers would be most representative of June in a May-June-July geomedian).

All up, I think this proposal sounds great - my only comment would be that if time_odc_to_ows and time_ows_to_odc must be inverses of each other by definition, could that logic be handled under the hood in OWS to save the user having to define a redundant func? (a minor issue!)

[SpacemanPaul]

if time_odc_to_ows and time_ows_to_odc must be inverses of each other by definition, could that logic be handled under the hood in OWS to save the user having to define a redundant func? (a minor issue!)

I can validate across indexed data to detect bad inverse implementations, but there's no way to do it under the hood without putting some pretty tight restrictions on what the function is allows to do. If you know a way to introspect a function and dynamically invert it I'm all ears!!

I agree it would be nice, but it's simply not possible.

[SpacemanPaul]

Unfortunately there's some technical debt in the form of the range tables which will need dealt with before this can be done. Will have to park this one a bit longer.

(Specifically the current inconsistency between the product range table being indexed by ODC product and multiproduct-range table being indexed by OWS layer.)