Build year aggregation

config/config.default.yaml

@@ -597,6 +597,7 @@ sector:
   overdimension_heat_generators:
     decentral: 1.1  #to cover demand peaks bigger than data
     central: 1.0
+  central_heat_everywhere: false
   chp: true
   micro_chp: false
   solar_thermal: true
@@ -669,6 +670,7 @@ sector:
   electricity_transmission_grid: true
   electricity_distribution_grid: true
   electricity_grid_connection: true
+  transmission_efficiency_enabled: true
   transmission_efficiency:
     DC:
       efficiency_static: 0.98
@@ -873,6 +875,10 @@ clustering:
   temporal:
     resolution_elec: false
     resolution_sector: false
+  build_year_aggregation:
+    enable: false
+    components: ["Generator", "Link", "Store", "StorageUnit"]
+    exclude_carriers: []
 
 # docs in https://pypsa-eur.readthedocs.io/en/latest/configuration.html#adjustments
 adjustments:

doc/configtables/clustering.csv

@@ -18,3 +18,6 @@ aggregation_strategies,,,
 temporal,,,Options for temporal resolution
 -- resolution_elec,--,"{false,``nH``; i.e. ``2H``-``6H``}","Resample the time-resolution by averaging over every ``n`` snapshots in :mod:`prepare_network`. **Warning:** This option should currently only be used with electricity-only networks, not for sector-coupled networks."
 -- resolution_sector,--,"{false,``nH``; i.e. ``2H``-``6H``}","Resample the time-resolution by averaging over every ``n`` snapshots in :mod:`prepare_sector_network`."
+build_year_aggregation,,,
+-- enable,bool,"{'true','false'}","Whether or not to aggregate similar components with different build years before myopic optimisations, and disaggregate after the optimisation."
+-- exclude_carriers,list,"List of Str like ['solar', 'onwind'] or empty list []","List of carriers for which components with different build years will not be aggregated. If empty, every component with different build years will be aggregated."

doc/configtables/sector-opts.csv

@@ -9,3 +9,4 @@ Trigger, Description, Definition, Status
 ``A``,Add agriculture sector,,In active use
 ``dist``+``n``,Add distribution grid with investment costs of ``n`` times costs in ``resources/costs_{cost_year}.csv``,,In active use
 ``seq``+``n``,Sets the CO2 sequestration potential to ``n`` Mt CO2 per year,,In active use
+``aggBuildYear``,Enable build year aggregation (see the ``build_year_aggregation`` option),,In active use

doc/configuration.rst

@@ -587,6 +587,9 @@ The list of available biomass is given by the category in `ENSPRESO_BIOMASS <htt
    ``feature:`` in ``simplify_network:``
    are only relevant if ``hac`` were chosen in ``algorithm``.
 
+.. note::
+   Build-year aggregation is an experimental feature which significantly reduces the memory-footprint of myopic foresight optimisations. In myopic foresight, there will be one copy of each component for every planning horizon; only the component for the "current" planning horizon will be extendable for each optimisation. By turning on build-year aggregation, these copies are merged into a single component right before the network is solved, and disaggregated again right after the network is solved. This feature should be tested to see if results are acceptable in any given context. The ```exclude_carriers`` should include ``DC``, and you should set the option ``central_heat_everywhere: true`` under the ``sector:`` configuration.
+
 .. tip::
    use ``min`` in ``p_nom_max:`` for more `
    conservative assumptions.

doc/foresight.rst

@@ -124,40 +124,25 @@ Grouping years indicates the bins limits for grouping the existing capacities of
 different technologies. Note that separate bins are defined for the power and
 heating plants due to different data sources.
 
-``grouping_years_power: [1980, 1985, 1990, 1995, 2000, 2005, 2010, 2015, 2020,
-2025, 2030]``
-
-``grouping_years_heat: [1980, 1985, 1990, 1995, 2000, 2005, 2010, 2015, 2019]``
-
-
-
-
-
-**threshold capacity**
-
 If for a technology, node, and grouping bin, the capacity is lower than
-threshold_capacity, it is ignored.
-
-``threshold_capacity: 10``
-
-
-
-
-**conventional carriers**
+``threshold_capacity`` (in MW), it is ignored.
 
 Conventional carriers indicate carriers used in the existing conventional
-technologies.
+technologies, allowing the selection of which kinds of existing capacities are to be added to the network and which to be left out. By default, all conventional technologies are included.
 
-    conventional_carriers:
+.. literalinclude:: ../config/config.default.yaml
+   :language: yaml
+   :start-at:   existing_capacities:
+   :end-before: # docs
 
-    \- lignite
 
-    \- coal
+**build year aggregation**
 
-    \- oil
+The ``build_year_aggregation`` option is found under the ``clustering`` section of the configuration file, and is set to ``false`` by default. (See also the relevant configuration documentation). By setting this option to ``true``, similar components with different build years aggregated before each optimisation (their capacities summed up, etc.) and disaggregated again after each optimisation. This can lead to drastic improvement in the memory footprint of the optimisations (up to approximately a factor of 5 in common use-cases), especially for optimisations at the later planning horizons.
 
-    \- uranium
+The aggregation has received some testing, and has lead to only small distortion in results. However, caution is advised in the use of this option. To test if it performs correctly in your use-case, use to ``aggBuildYear`` flag in the ``{sector_opts}`` wildcard to run a version of your model with and without build year aggregation, and see if the results are the same.
 
+Build year aggregation has been tested with the default configuration, with one exception: you must set ``central_heat_everywhere`` to ``true`` under the ``sector:`` configuration section. Otherwise one might get links (with waste heat) which are connected to central heating in some planning horizons and not in others. Alternatively, turn off all waste heat modelling.
 
 
 

doc/release_notes.rst

@@ -78,6 +78,8 @@ Upcoming Release
 
 * Bugfix: demand for ammonia was double-counted at current/near-term planning horizons when ``sector['ammonia']`` was set to ``True``.
 
+* Add an experimental feature to aggregate components by build-year before optimisation, leading to a substantial reduction in memory footprint for myopic foresight optimisations. This aggregation can lead to minor changes in results, and the feature should be tested in the intended context before being used.
+
 PyPSA-Eur 0.13.0 (13th September 2024)
 ======================================
 

rules/solve_myopic.smk

@@ -108,6 +108,7 @@ rule solve_sector_network_myopic:
     params:
         solving=config_provider("solving"),
         foresight=config_provider("foresight"),
+        build_year_agg=config_provider("clustering", "build_year_aggregation"),
         planning_horizons=config_provider("scenario", "planning_horizons"),
         co2_sequestration_potential=config_provider(
             "sector", "co2_sequestration_potential", default=200

scripts/_helpers.py

@@ -738,6 +738,9 @@ def update_config_from_wildcards(config, w, inplace=True):
         if seq_limit is not None:
             config["sector"]["co2_sequestration_potential"] = seq_limit
 
+        if "aggBuildYear" in opts:
+            config["clustering"]["build_year_aggregation"]["enable"] = True
+
         # any config option can be represented in wildcard
         for o in opts:
             if o.startswith("CF+"):

scripts/add_existing_baseyear.py

@@ -276,7 +276,7 @@ def add_power_capacities_installed_before_baseyear(n, grouping_years, costs, bas
                     new_capacity.index,
                     suffix=name_suffix,
                     bus=new_capacity.index,
-                    carrier=generator,
+                    carrier=generator + suffix,
                     p_nom=new_capacity,
                     marginal_cost=marginal_cost,
                     capital_cost=capital_cost,

scripts/make_summary.py

@@ -293,7 +293,7 @@ def calculate_energy(n, label, energy):
                     .sum()
                 )
                 # remove values where bus is missing (bug in nomopyomo)
-                no_bus = c.df.index[c.df["bus" + port] == ""]
+                no_bus = c.df.index[c.df["bus" + port] == ""].intersection(totals.index)
                 totals.loc[no_bus] = float(
                     n.component_attrs[c.name].loc["p" + port, "default"]
                 )
@@ -347,7 +347,8 @@ def calculate_supply(n, label, supply):
 
                 # lots of sign compensation for direction and to do maximums
                 s = (-1) ** (1 - int(end)) * (
-                    (-1) ** int(end) * c.pnl["p" + end][items]
+                    (-1) ** int(end)
+                    * c.pnl["p" + end].reindex(columns=items, fill_value=0.0)
                 ).max().groupby(c.df.loc[items, "carrier"]).sum()
                 s.index = s.index + end
                 s = pd.concat([s], keys=[c.list_name])
@@ -397,9 +398,11 @@ def calculate_supply_energy(n, label, supply_energy):
                 if len(items) == 0:
                     continue
 
-                s = (-1) * c.pnl["p" + end][items].multiply(
-                    n.snapshot_weightings.generators, axis=0
-                ).sum().groupby(c.df.loc[items, "carrier"]).sum()
+                s = (-1) * c.pnl["p" + end].reindex(
+                    columns=items, fill_value=0.0
+                ).multiply(n.snapshot_weightings.generators, axis=0).sum().groupby(
+                    c.df.loc[items, "carrier"]
+                ).sum()
                 s.index = s.index + end
                 s = pd.concat([s], keys=[c.list_name])
                 s = pd.concat([s], keys=[i])

scripts/prepare_sector_network.py

@@ -2090,7 +2090,9 @@ def add_heat(n: pypsa.Network, costs: pd.DataFrame, cop: xr.DataArray):
         overdim_factor = options["overdimension_heat_generators"][
             heat_system.central_or_decentral
         ]
-        if heat_system == HeatSystem.URBAN_CENTRAL:
+        if (heat_system == HeatSystem.URBAN_CENTRAL) and not options[
+            "central_heat_everywhere"
+        ]:
             nodes = dist_fraction.index[dist_fraction > 0]
         else:
             nodes = pop_layout.index
@@ -4680,8 +4682,9 @@ def add_enhanced_geothermal(n, egs_potentials, egs_overlap, costs):
     if options["electricity_grid_connection"]:
         add_electricity_grid_connection(n, costs)
 
-    for k, v in options["transmission_efficiency"].items():
-        lossy_bidirectional_links(n, k, v)
+    if options.get("transmission_efficiency_enabled", True):
+        for k, v in options["transmission_efficiency"].items():
+            lossy_bidirectional_links(n, k, v)
 
     # Workaround: Remove lines with conflicting (and unrealistic) properties
     # cf. https://github.com/PyPSA/pypsa-eur/issues/444