Enable sub-national Eurostat data to be applied in a shape-agnostic fashion

[brynpickering]

Datasets are rarely available with the same resolution as we need in Euro-Calliope layers. Instead, we distribute e.g. national data to sub-national regions using datasets like population. There is much scope to increase the accuracy of this regionalisation, using various datasets from Eurostat. But, all these datasets need to be applicable at any layer resolution. E.g. if using arbitrary shapefiles as the layer resolution (#76), it should still be possible to use a NUTS3 resolved dataset to regionalise demand/supply data.

Shape-agnostic regionalisation is / will be possible for industry subsectors relying on the EU-ETS (#63) and on demand based on population (#77), but these new sources from Eurostat need a helper function to enable their normalisation and rasterisation. This includes combining datasets iteratively (e.g. Employment data gets us down to NUT2 level for industry, but freight data can then be used to get us down to a NUTS3 level. Employment data is worthwhile as it better explains known sub-national distributions).

Datasets this applies to include: Eurostat freight loading/unloading (industry subsectors), Gross value added (commercial), and freight loading/unloading (industry subsectors), See here for the current application in our sector-coupled sibling.

In addition, aviation demand could be disaggregated on airports and the quantity of passengers travelling through them. Shipping could be similarly disaggregated based on ports. At present, neither of these is implemented (no known data source), and these are distributed the same as industry subsectors.

EDIT: this issue no longer discusses configurability of regionalisation of data.

[brynpickering]

This links to and expands the scope of bringing the relative residential/industrial electricity demand from solar-and-wind-potentials, as discussed in #63

[timtroendle]

What exactly do we need to do here? What I see is:

1. Allow for custom shapes,
2. Integrate renewable potentials (handled in Enable solar-and-wind-potentials to be a submodule #77),
3. Integrate industry demand (handled in Improve industrial electricity demand calculation #63).

So am I right in saying that this issue is solved when (1) is solved? If not, what else?

[brynpickering]

There's a few other things in addition. This includes regionalising service-sector demand (e.g. commercial building heat demand) based on service-sector gross-added value (this isn't industry demand) and (possibly) regionalising marine and aviation fuel demand based on activity of airports and shipping ports (I don't do this at present, but use industry demand as a proxy for possible fuel generation hot-spots). Allowing for custom shapes (1) isn't strictly necessary here, but rather that this process should be shape-agnostic, as it is with the current population/EU-ETS method.

[timtroendle]

I don't understand what this issue is about. This workflow is shape-agnostic, apart from the three points mentioned above. When we add things to this workflow, those things should of course be shape-agnostic.

[brynpickering]

Yes, all things should be added in a shape-agnostic fashion to the workflow. However, they partially aren't in the sector-coupled sibling, hence the initial driver for me to open this issue in preparation for the upcoming code sprint. But it's also a question of how we expose this regionalisation to the user. One could regionalise everything based on population, or use only Eurostat employment data (and not EU-ETS) to regionalise industry sub-sectors, etc. Regionalisation could be baked in or it could be configurable, which is the second component of this issue.

I suspect we don't care about the configurability, so I will update this issue to focus only on the need to handle rasterisation and combination of Eurostat data (which comes in at NUTS2-3 resolution) for use in regionalisation.

[brynpickering]

@timtroendle does this now make more sense?

[timtroendle]

Yes. Is it correct to say that we do not have an explicit "Allow for custom shapes" issue at this point?

[brynpickering]

Yes. Is it correct to say that we do not have an explicit "Allow for custom shapes" issue at this point?

#76

[sjpfenninger]

Comment from Bryn's handover document:

"This can be done using a geopandas overlay between the spatial datasets that are already included and the geographical units of the resolution of interest. The key here is having a way to subnationalise a particular bit of data on the fly. This requires configuration options to map subsectors to distribution methods (e.g. household building heat: population, commercial building heat: gva, industry steel: EU-ETS, industry textiles and leather: freight & employees) so that if you ask for the subnational data on one of them it can do things automatically. However, it would also ideally only do the overlay of one spatial dataset with one model resolution once, since the operation can be costly. So perhaps first there needs to be this overlay and then that is saved to file and used by all future rules that need it."