Merge pull request #129 from PyPSA/database-pypsa-de

Preparation for public model Pypsa-DE

README.md

@@ -1,6 +1,6 @@
-# Kopernikus-Projekt Ariadne - Gesamtsystemmodell PyPSA-Eur
+# Kopernikus-Projekt Ariadne - Gesamtsystemmodell PyPSA-DE
 
-Dieses Repository enthält das Gesamtsystemmodell für das Kopernikus-Projekt Ariadne, basierend auf der Toolbox PyPSA und dem Datensatz PyPSA-Eur. Das Modell bildet Deutschland mit hoher geographischer Auflösung, mit voller Sektorenkopplung und mit Integration in das europäische Energiesystem ab.
+Dieses Repository enthält das Gesamtsystemmodell PyPSA-DE für das Kopernikus-Projekt Ariadne, basierend auf der Toolbox PyPSA und dem Datensatz PyPSA-Eur. Das Modell bildet Deutschland mit hoher geographischer Auflösung, mit voller Sektorenkopplung und mit Integration in das europäische Energiesystem ab.
 
 This repository contains the entire scientific project, including data sources and code. The philosophy behind this repository is that no intermediary results are included, but all results are computed from raw data and code.
 
@@ -24,32 +24,45 @@ More information on Git Submodules can be found [here](https://git-scm.com/book/
 
 ## Getting ready
 
-You need [mamba](https://mamba.readthedocs.io/en/latest/) to run the analysis. Using mamba, you can create an environment from within you can run it:
+You need conda or [mamba](https://mamba.readthedocs.io/en/latest/) to run the analysis. Using mamba, you can create an environment from within you can run it:
 
     mamba env create -f environment.yaml
 
-## Provide login details
+## For external users: Use config.public.yaml
 
-The snakemake rule `retrieve_ariadne_database` logs into the IIASA Database via the [`pyam`](https://pyam-iamc.readthedocs.io/en/stable/tutorials/iiasa.html) package. The credentials for logging into this database have to be stored locally on your machine with `ixmp4`. To do this, run
+The default workflow configured for this repository assumes access to the internal Ariadne2 database. Users that do not have the required login details can run the analysis based on the data published during the [first phase of the Ariadne project](https://data.ece.iiasa.ac.at/ariadne/).
+
+This is possible by providing an additional config to the snakemake workflow. For every `snakemake COMMAND` specified in the instructions below, public users should use:
+
+```
+snakemake --configfile=config/config.public.yaml COMMAND
+```
+
+The additional config file specifies the required database, model, and scenario names for Ariadne1. If public users wish to edit the default scenario specifications, they should change `scenarios.public.yaml` instead of `scenarios.manual.yaml`. More details on using scenarios are given below.
+
+## For internal users: Provide login details
+
+The snakemake rule `retrieve_ariadne_database` logs into the interal Ariadne IIASA Database via the [`pyam`](https://pyam-iamc.readthedocs.io/en/stable/tutorials/iiasa.html) package. The credentials for logging into this database have to be stored locally on your machine with `ixmp4`. To do this activate the project environment and run
 
 ```
 ixmp4 login <username>
 ```
 
-You will be prompted to enter your `<password>`. 
+You will be prompted to enter your `<password>`.
 
 Caveat: These credentials are stored on your machine in plain text.
 
+To switch between internal and public use, the command `ixmp4 logout` may be necessary.
+
 ## Run the analysis
 
-Before running any scenarios, the rule `build_scenarios` must be executed. This will write the file `config/scenarios.automated.yaml` which includes transport shares and ksg goals from the iiasa database as well as the information from the file `config/scenarios.manual.yaml`.
+Before running any analysis with scenarios, the rule `build_scenarios` must be executed. This will create the file `config/scenarios.automated.yaml` which includes input data and CO2 targets from the IIASA Ariadne database as well as the specifications from the manual scenario file. [This file is specified in the default config.yaml via they key `run:scenarios:manual_file` (by default located at `config/scenarios.manual.yaml`)].
 
     snakemake -call build_scenarios -f
 
-Note that the hierarchy of scenario files is the following: `scenarios.automated.yaml` > `config.yaml` > `config.default.yaml`
-Changes in the file `scenarios.manual.yaml` are only taken into account if the rule `build_scenarios` is executed.
+Note that the hierarchy of scenario files is the following: `scenarios.automated.yaml` > (any `explicitly specified --configfiles`) > `config.yaml `> `config.default.yaml `Changes in the file `scenarios.manual.yaml `are only taken into account if the rule `build_scenarios` is executed.
 
-For the first run open config.yaml and set
+For the first run, open config.yaml and set
 
     enable:
         retrieve: true # set to false once initial data is retrieved
@@ -67,12 +80,31 @@ To generate a PDF of the dependency graph of all steps `build/dag.pdf` run:
 
 ## Repo structure
 
-* `config`: configurations used in the study
-* `cutouts`: very large weather data cutouts supplied by atlite library
-* `data`: place for raw data
-* `resources`: place for intermediate/processing data for the workflow
+* `config`: configuration files
+* `ariadne-data`: Germany specific data from the Ariadne project
+* `workflow`: contains the Snakemake workflow, including the submodule PyPSA-Eur and specific scripts for Germany
+* `cutouts`: very large weather data cutouts supplied by atlite library (does not exist initially)
+* `data`: place for raw data (does not exist initially)
+* `resources`: place for intermediate/processing data for the workflow (does not exist initially)
 * `results`: will contain all results (does not exist initially)
-* `workflow`: contains the Snakemake workflow, including the submodule PyPSA-Eur
+
+## Differences to PyPSA-EUR
+
+- Specific cost assumption for Germany:
+  - Gas, Oil, Coal prices
+  - electrolysis and heat-pump costs
+  - Infrastructure costs according to the Netzentwicklungsplan 23 (NEP23)
+  - option for pessimstic, mean and optimistic cost development
+- Transport and Industry demands as well as heating stock imported from the sectoral models in the Ariadne consortium
+- More detailed data on CHPs in Germany
+- Option for building the German Wasserstoffkernnetz
+- The model has been validated against 2020 electricity data for Germany
+- National CO2-Targets according to the Klimaschutzgesetz
+- Additional constraints that limit maximum capacity of specific technologies
+- Import constraints
+- Renewable build out according to the Wind-an-Land, Wind-auf-See and Solarstrategie laws
+- A comprehensive reporting  module that exports Capacity Expansion, Primary/Secondary/Final Energy, CO2 Emissions per Sector, Trade, Investments, ...
+- Plotting functionality to compare different scenarios
 
 ## License
 

config/config.public.yaml

@@ -0,0 +1,35 @@
+# SPDX-FileCopyrightText: : 2017-2024 The PyPSA-Eur Authors
+#
+# SPDX-License-Identifier: CC0-1.0
+
+run:
+  prefix: 20240627public_db
+  name:
+  - 8Gt_Bal_v3
+  # - 8Gt_Elec_v3
+  # - 8Gt_H2_v3
+  scenarios:
+    enable: true
+    manual_file: config/scenarios.public.yaml
+    file: config/scenarios.automated.yaml
+  shared_resources: 
+    policy: base #stops recalculating
+    exclude:
+      - existing_heating.csv # specify files which should not be shared between scenarios
+      - costs
+      - retrieve_cost # This is necessary to save retrieve_cost_data_{year}.log in the correct folder
+
+iiasa_database:
+  db_name: ariadne
+  leitmodelle:
+    general: REMIND-EU v1.1
+    buildings: REMod v1.0
+    transport: DLR
+    transport_stock: DLR-VECTOR21
+    industry: FORECAST v1.0
+  scenarios:
+    - 8Gt_Bal_v3
+    - 8Gt_Elec_v3
+    - 8Gt_H2_v3
+  reference_scenario: 8Gt_Bal_v3
+  region: Deutschland

config/config.yaml

@@ -16,6 +16,7 @@ run:
   # - KN2045minus_SupplyFocus
   scenarios:
     enable: true
+    manual_file: config/scenarios.manual.yaml
     file: config/scenarios.automated.yaml
   shared_resources: 
     policy: base #stops recalculating
@@ -33,10 +34,10 @@ run:
 iiasa_database:
   db_name: ariadne2_intern
   leitmodelle:
-    - REMIND-EU v1.1
-    - REMod v1.0
-    - Aladin v1
-    - FORECAST v1.0
+    general: REMIND-EU v1.1
+    buildings: REMod v1.0
+    transport: Aladin v1
+    industry: FORECAST v1.0
   scenarios:
     - CurrentPolicies
     - KN2045_Elec_v4

config/scenarios.public.yaml

@@ -0,0 +1,174 @@
+# -*- coding: utf-8 -*-
+# SPDX-FileCopyrightText: : 2017-2023 The PyPSA-Eur Authors
+#
+# SPDX-License-Identifier: MIT
+
+
+8Gt_Bal_v3:
+  iiasa_database:
+    reference_scenario: 8Gt_Bal_v3
+    fallback_reference_scenario: 8Gt_Bal_v3
+  co2_budget_DE_source: KSG
+
+  costs:
+    horizon: "mean"
+    NEP: 2023
+    transmission: "overhead" # either overhead line ("overhead") or underground cable ("underground")
+
+  # boundary condition of maximum volumes
+  limits_volume_max:
+    # constrain electricity import in TWh
+    electricity_import:
+      DE:
+        2020: -20
+        2025: 0
+        2030: 0
+        2035: 40
+        2040: 80
+        2045: 125
+    electrolysis:
+      DE:
+        2020: 0
+        2025: 5
+        2030: 45
+        2035: 130
+        2040: 215
+        2045: 300
+    h2_derivate_import:
+      DE:
+        2020: 0
+        2025: 0
+        2030: 10
+        2035: 105
+        2040: 200
+        2045: 300
+    h2_import:
+      DE:
+        2020: 0
+        2025: 5
+        2030: 15
+        2035: 115
+        2040: 220
+        2045: 325
+  limits_volume_min:
+    electrolysis:
+      DE:
+        2020: 0
+        2025: 0
+        2030: 0
+        2035: 0
+        2040: 0
+        2045: 0
+
+8Gt_Elec_v3:
+  iiasa_database:
+    reference_scenario: 8Gt_Elec_v3
+    fallback_reference_scenario: 8Gt_Elec_v3
+  co2_budget_DE_source: KSG
+
+  costs:
+    horizon: "mean"
+    NEP: 2023
+    transmission: "overhead" # either overhead line ("overhead") or underground cable ("underground")
+
+  limits_volume_max:
+    # constrain electricity import in TWh
+    electricity_import:
+      DE:
+        2020: -20
+        2025: 0
+        2030: 0
+        2035: 50
+        2040: 100
+        2045: 150
+    electrolysis:
+      DE:
+        2020: 0
+        2025: 5
+        2030: 45
+        2035: 95
+        2040: 145
+        2045: 200
+    h2_derivate_import:
+      DE:
+        2020: 0
+        2025: 0
+        2030: 10
+        2035: 70
+        2040: 130
+        2045: 200
+    h2_import:
+      DE:
+        2020: 0
+        2025: 5
+        2030: 10
+        2035: 90
+        2040: 170
+        2045: 250
+  limits_volume_min:
+    electrolysis:
+      DE:
+        2020: 0
+        2025: 0
+        2030: 0
+        2035: 0
+        2040: 0
+        2045: 0
+
+8Gt_H2_v3:
+  iiasa_database:
+    reference_scenario: 8Gt_H2_v3
+    fallback_reference_scenario: 8Gt_H2_v3
+  co2_budget_DE_source: KSG
+
+  costs:
+    horizon: "mean"
+    NEP: 2023
+    transmission: "overhead" # either overhead line ("overhead") or underground cable ("underground")
+
+  limits_volume_max:
+    # constrain electricity import in TWh
+    electricity_import:
+      DE:
+        2020: -20
+        2025: 0
+        2030: 0
+        2035: 30
+        2040: 70
+        2045: 100 # scenario guidelines
+
+    # constrain hydrogen import in TWh
+    h2_import:
+      DE:
+        2020: 0
+        2025: 5
+        2030: 45 # scenario guidelines 
+        2035: 155
+        2040: 265
+        2045: 400 # scenario guidelines
+    # import of h2 derivatives in TWh
+    h2_derivate_import:
+      DE:
+        2020: 0
+        2025: 0
+        2030: 10 # scenario guidelines
+        2035: 140
+        2040: 270
+        2045: 400 # scenario guidelines
+    electrolysis:
+      DE:
+        2020: 0
+        2025: 5
+        2030: 45 # scenario guidelines
+        2035: 160
+        2040: 275
+        2045: 400 # scenario guidelines
+
+  limits_volume_min:
+    electrolysis:
+      DE:
+        2025: 0
+        2030: 0
+        2035: 0
+        2040: 0
+        2045: 200

workflow/Snakefile

@@ -132,8 +132,10 @@ if config["enable"]["retrieve"] and config["enable"].get("retrieve_cost_data", T
 
 rule build_mobility_demand:
     params:
+        db_name=config_provider("iiasa_database", "db_name"),
         reference_scenario=config_provider("iiasa_database", "reference_scenario"),
         planning_horizons=config_provider("scenario", "planning_horizons"),
+        leitmodelle=config_provider("iiasa_database", "leitmodelle"),
     input:
         ariadne=resources("ariadne_database.csv"),
         clustered_pop_layout=resources("pop_layout_elec_s{simpl}_{clusters}.csv"),
@@ -245,6 +247,8 @@ use rule build_existing_heating_distribution from pypsaeur with:
         existing_heating=resources("existing_heating.csv"),
 
 rule modify_industry_demand:
+    params:
+        db_name=config_provider("iiasa_database", "db_name"),
     input:
         ariadne=resources("ariadne_database.csv"),
         industrial_production_per_country_tomorrow=resources("industrial_production_per_country_tomorrow_{planning_horizons}.csv"),
@@ -418,9 +422,11 @@ rule ariadne_all:
 rule build_scenarios:
     params:
         scenarios=config_provider("run", "name"),
+        db_name=config_provider("iiasa_database","db_name"),
+        leitmodelle=config_provider("iiasa_database", "leitmodelle"),
     input:
         ariadne_database=resources("ariadne_database.csv"),
-        scenario_yaml="config/scenarios.manual.yaml",
+        scenario_yaml=config["run"]["scenarios"]["manual_file"],
     output:
         scenario_yaml=config["run"]["scenarios"]["file"],
     log: