Draft modular config structure

[ekatef]

An attempt to organise the configuration file in a readable way

Changes proposed in this Pull Request

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

That is indeed a great draft. Clearly this is not (yet) integrated as a workflow, but it is a great proposal.
It will have implications to the other repo (-distribution and alike), but it is a great proposal in my opinion, and we need to start :)

We can reflect on the naming, for example:

• geometry may be replaced with spatial
• weather may be replaced with load_weather or alike; just to clarify also the load option.

They are quite many files; I am wondering if the sector configs may be merged (e.g. heating/hydrogen/transport) into a config.sector.yaml for example

Moreover, I'm wondering about the "main" settings of the scenario, run and enable keys for example; maybe a "header" config may be interesting to aggregate some of those options?

It would be great to collect ideas

[ekatef]

Thank you so much for looking into that @davide-f! Happy to hear that you like it 🙂

I expect that it'll take some iterations to find a convenient and robust approach to deal with the configs. But it feels easier to think about different options when playing with a possible implementation. So, let's experiment with that!

Agree that it would be cleaner to keep a reasonable number of the modular configs. Have renamed geometry with spatial and merged:

• heating+transport+H2 into the sectoral config
• monte-carlo parameters included into solving config.

Not sure it would increase clarity if we add load to weather in the file name. Currently, mixing the weather year with socioecomonic parameters seems to be a reason of many confusions. So, my preference would be to leave weather only, but happy to revise if you think it'll make things more clear

Absolutely agree that we need also to think about management of the configuration files, also keeping in mind the requirements of the distribution model.

A great point on having a header file. Happy to implement that and have to find a proper way to do so 😄 That directly relates to creating a good architecture, and I'd be very grateful for your support in that.

[ekatef]

Some additionally points which would be great to discuss are:

• Would be great to ensure that management of the configs do not lead to any data loss. In particular, if config.yaml presents in the working folder, it must be kept untouched (can we do a reserve copy?)
• Some sanity checks of the merged config.yaml are needed. E.g. according to this issue Snakemake allows for duplicated keys in the config.

Obviously, more ideas are welcome

[ekatef]

As discussed previously, it would be also great to add a parser for the config to check it for consistency.

The requirements can include:

• all the keys are unique
• years used to define different parameters are the same (not realistic, probably) or there is an overview on which year is assumed for what (e.g. GDP, installed renewable capacity, industry capacity, energy balances, weather year)
• parameters of the sector-coupled scenarios are consistent (e.g. bev_avail_max > bev_avail_mean)
• assumptions for particular scenarios can be easily checked(is also linked with the need to document the configuration parameters for the sector-coupled capabilities Add sector-coupled configuration to docs #1208)

[ekatef]

The basic functionality is there and can be improved.

TODOs

Documentation

• update configuration.rst to capture the changed structure of all the configuration parameters
• add configuration_base.rst to describe the (shortened) list of parameters included into config.default.yaml
• add diagrams for major scenario parameters of sector-coupled part

Functionality

• allow to specify only a single name of the cutout in config.default.yaml which should overwrite all the cutout names available in the configs
• check for non-ambiguity and fix issues if needed (the config parameters which should have a single value should not have multiple items which can easily happen if merging of the configs went wrong)

[ekatef]

@energyLS could you please help with the definitions of the parameters of the sector-coupled part? 🙏🏽 In particular, I'd be very happy to understand:

• how do we use the parameters which sweep trough different Co2 reduction rates, such as land_transport_electric_share?
• what is the meaning of abbreviations in the scenario names, e.g. BU_2030 or AB_2030 in land_transport_fuel_cell_share?
• what is applicability of the heating-related parameters?

My feeling is that the revised config.default looks quite appropriate, and hopefully we are getting closer to finalise it 🙂

[energyLS]

@energyLS could you please help with the definitions of the parameters of the sector-coupled part? 🙏🏽 In particular, I'd be very happy to understand:

* how do we use the parameters which sweep trough different Co2 reduction rates, such as  [land_transport_electric_share](https://github.com/pypsa-meets-earth/pypsa-earth/blob/9a8bc415ca450461622f05829b150d63d8a0e8dc/config.default.yaml#L616-L628)?

* what is the meaning of abbreviations in the scenario names, e.g. `BU_2030` or `AB_2030` in [land_transport_fuel_cell_share](https://github.com/pypsa-meets-earth/pypsa-earth/blob/9a8bc415ca450461622f05829b150d63d8a0e8dc/config.default.yaml#L643-L652)?

* what is applicability of the [heating-related parameters](https://github.com/pypsa-meets-earth/pypsa-earth/blob/9a8bc415ca450461622f05829b150d63d8a0e8dc/config.default.yaml#L540-L561)?

My feeling is that the revised config.default looks quite appropriate, and hopefully we are getting closer to finalise it 🙂

@ekatef happy to support here:

• If the "dynamic" option is enabled, then the land transport is linked to the co2 reduction. This means, that e.g. if a co2 reduction of 20% is applied (via CO2L0.80), then it will use the respective land transport share accordingly. This is e.g. useful, if you perform a parameter sweep through the co2 reduction the land transport is adjusted automatically. If the option is disabled, the standard parameters are applied.
• The scenario names are linked to the demand scenario, and the respective parames are then used. e.g. if you select scenario "AB", then AB_2030 will be selected. For more details on the scenarios, I think @hazemakhalek can help you out
• In theory, it defines the share of district heating and it's losses, in addition to space heating demand reductions through e.g. building renovation etc. However, I am not sure to what degree this implementation is used. I am aware of this input data but I am not deep into how it works.

I hope this helps, just let me know if there are further questions :) Thank you!

[ekatef]

@ekatef happy to support here:

• If the "dynamic" option is enabled, then the land transport is linked to the co2 reduction. This means, that e.g. if a co2 reduction of 20% is applied (via CO2L0.80), then it will use the respective land transport share accordingly. This is e.g. useful, if you perform a parameter sweep through the co2 reduction the land transport is adjusted automatically. If the option is disabled, the standard parameters are applied.
• The scenario names are linked to the demand scenario, and the respective parames are then used. e.g. if you select scenario "AB", then AB_2030 will be selected. For more details on the scenarios, I think @hazemakhalek can help you out
• In theory, it defines the share of district heating and it's losses, in addition to space heating demand reductions through e.g. building renovation etc. However, I am not sure to what degree this implementation is used. I am aware of this input data but I am not deep into how it works.

I hope this helps, just let me know if there are further questions :) Thank you!

@energyLS perfect, thanks a lot for the comments! It helps a lot 🙂
I'll try to quickly sketch some minimalistic documentation on this part which we would be able to improve during the next iterations.