Remove get_formatted_array

changelog.rst

@@ -20,6 +20,8 @@ User-facing changes
 
 |changed| |backwards-incompatible| The `loc::tech` and `loc::tech::carrier` sets have been removed. Model components are now indexed separately over `node`, `tech`, and `carrier` (where applicable). Although primarily an internal change, this affects the xarray dataset structure and hence how users access data in `model.inputs` and `model.results`. For example, `model.inputs.energy_cap_max.loc[{"loc_techs": "X:pv"}]` in v0.6 needs to be changed to `model.inputs.energy_cap_max.loc[{"nodes": "X", "techs": "pv"}]` in v0.7. This is functionally equivalent to first calling `model.get_formatted_array("energy_cap_max")` in v0.6, which is no longer necessary in v0.7.
 
+|changed| |backwards-incompatible| `get_formatted_array` has been removed in favour of directly accessing the data, since we no longer concatenate sets. E.g., `model.get_formatted_array("storage")` -> `model.results.storage`.
+
 |changed| |backwards-incompatible| The dimensions of the model data no longer include all possible subsets. E.g. a user can no longer access `loc_techs_supply` to view the location/technology pairs which have defined `supply` as their top-level parent. Instead, the same subset can be supplied by calling `model.inputs.inheritance.str.endswith('supply')` to create a boolean array of technologies with `supply` as their top-level parent.
 
 |changed| |backwards-incompatible| Group constraints have been removed. They will be replaced by `custom constraint` functionality.

doc/_static/notebooks/calliope_model_object.ipynb

@@ -157,7 +157,7 @@
     }
    ],
    "source": [
-    "# This includes location-specific overrides, such as energy_cap_max of 50 for the pv technology at location X3\n",
+    "# This includes location-specific overrides, such as flow_cap_max of 50 for the pv technology at location X3\n",
     "\n",
     "m._model_run[\"nodes\"][\"X3\"][\"techs\"][\"pv\"]"
    ]
@@ -5279,7 +5279,7 @@
    "source": [
     "# Data can also be reformatted to be easier to read (removes dimension concatenation).\n",
     "# Conversion to a pandas DataFrame is a good idea for greater readibility.\n",
-    "m.get_formatted_array(\"energy_cap\").to_pandas()"
+    "m.results.energy_cap.to_pandas()"
    ]
   },
   {
@@ -5313,7 +5313,9 @@
    "source": [
     "# >2 dimensions cannot be easily viewed in a pandas dataframe, unless a MultiIndex is used.\n",
     "# To view a 4-dimensional result, we can use `to_series()`\n",
-    "m.results(\"carrier_prod\").to_series().dropna()  # drop_na() removes all NaN values"
+    "\n",
+    "# drop_na() removes all NaN values\n",
+    "m.results.flow_out.to_series().dropna()"
    ]
   },
   {

doc/_static/notebooks/milp.ipynb

@@ -2084,7 +2084,7 @@
    ],
    "source": [
     "# We can sum operating units of CHP over all locations and turn the result into a pandas DataFrame\n",
-    "df_units = model.get_formatted_array(\"operating_units\").sum(\"nodes\").to_series()\n",
+    "df_units = model.results.operating_units.sum(\"nodes\").to_series()\n",
     "\n",
     "# The information about the dataframe tells us about the amount of data it holds in the index and in each column\n",
     "df_units.info()"

doc/_static/notebooks/national_scale.ipynb

@@ -1273,9 +1273,8 @@
     }
    ],
    "source": [
-    "# To reformat the array, deconcatenating loc_techs / loc_tech_carriers, you can use model.get_formatted_array()\n",
-    "# You can then apply loc/tech/carrier only operations, like summing information over locations:\n",
-    "model.get_formatted_array(\"resource\").sum(\"nodes\").to_series().unstack(\"techs\")"
+    "# You can apply node/tech/carrier only operations, like summing information over locations:\n",
+    "model.inputs.resource.sum(\"nodes\").to_series().unstack(\"techs\")"
    ]
   },
   {
@@ -2366,12 +2365,7 @@
    ],
    "source": [
     "# We can sum power output over all locations and turn the result into a pandas DataFrame\n",
-    "df_power = (\n",
-    "    model.get_formatted_array(\"carrier_prod\")\n",
-    "    .loc[{\"carriers\": \"power\"}]\n",
-    "    .sum(\"nodes\")\n",
-    "    .to_series()\n",
-    ")\n",
+    "df_power = model.results.flow_out.loc[{\"carriers\": \"power\"}].sum(\"nodes\").to_series()\n",
     "\n",
     "# The information about the dataframe tells us about the amount of data it holds in the index and in each column\n",
     "df_power.info()"
@@ -8310,9 +8304,7 @@
     "# notice all the NaN values which appear when seperating loc::techs to locs and techs.\n",
     "# Any NaN value means we never considered that combination of `loc` and `tech` for the variable\n",
     "\n",
-    "costs = (\n",
-    "    model.get_formatted_array(\"cost\").loc[{\"costs\": \"monetary\"}].to_series().dropna()\n",
-    ")\n",
+    "costs = model.results.cost.loc[{\"costs\": \"monetary\"}].to_series().dropna()\n",
     "costs"
    ]
   },

doc/_static/notebooks/urban_scale.ipynb

@@ -1476,9 +1476,8 @@
     }
    ],
    "source": [
-    "# To reformat the array, deconcatenating loc_techs / loc_tech_carriers, you can use model.get_formatted_array()\n",
-    "# You can then apply loc/tech/carrier only operations, like summing information over locations:\n",
-    "model.get_formatted_array(\"resource\").sum(\"locs\").to_pandas()"
+    "# You can apply node/tech/carrier only operations, like summing information over locations:\n",
+    "model.inputs.source_max.sum(\"nodes\").to_pandas()"
    ]
   },
   {
@@ -2631,13 +2630,7 @@
    ],
    "source": [
     "# We can sum heat output over all locations and turn the result into a pandas DataFrame\n",
-    "df_heat = (\n",
-    "    model.get_formatted_array(\"carrier_prod\")\n",
-    "    .loc[{\"carriers\": \"heat\"}]\n",
-    "    .sum(\"locs\")\n",
-    "    .to_pandas()\n",
-    "    .T\n",
-    ")\n",
+    "df_heat = model.results.flow_out.loc[{\"carriers\": \"heat\"}].sum(\"locs\").to_pandas().T\n",
     "\n",
     "# The information about the dataframe tells us about the amount of data it holds in the index and in each column\n",
     "df_heat.info()"
@@ -7166,7 +7159,7 @@
     "# notice all the NaN values which appear when seperating loc::techs to locs and techs.\n",
     "# Any NaN value means we never considered that combination of `loc` and `tech` for the variable\n",
     "\n",
-    "costs = model.get_formatted_array(\"cost\").loc[{\"costs\": \"monetary\"}].to_pandas()\n",
+    "costs = model.results.cost.loc[{\"costs\": \"monetary\"}].to_pandas()\n",
     "costs"
    ]
   },

doc/helpers/generate_math.py

@@ -80,7 +80,7 @@ def generate_model_config() -> dict[str, dict]:
     dummy_techs = {
         "demand_tech": {
             "essentials": {"parent": "demand", "carrier": "electricity"},
-            "constraints": {"resource": "df=ts_neg"},
+            "constraints": {"sink_equals": "df=ts"},
         }
     }
 

doc/user/advanced_constraints.rst

@@ -23,7 +23,7 @@ An example use of ``supply_plus`` is to define a concentrating solar power (CSP)
 
 See the :ref:`listing of supply_plus configuration <abstract_base_tech_definitions>` in the abstract base tech group definitions for the additional constraints that are possible.
 
-.. Warning:: When analysing results from supply_plus, care must be taken to correctly account for the losses along the transformation from resource to carrier. For example, charging of storage from the resource may have a ``resource_eff``-associated loss with it, while discharging storage to produce the carrier may have a different loss resulting from a combination of ``energy_eff`` and ``parasitic_eff``. Such intermediate conversion losses need to be kept in mind when comparing discharge from storage with ``carrier_prod`` in the same time step.
+.. Warning:: When analysing results from supply_plus, care must be taken to correctly account for the losses along the transformation from resource to carrier. For example, charging of storage from the resource may have a ``source_eff``-associated loss with it, while discharging storage to produce the carrier may have a different loss resulting from a combination of ``flow_eff`` and ``parasitic_eff``. Such intermediate conversion losses need to be kept in mind when comparing discharge from storage with ``flow_out`` in the same time step.
 
 .. _conversion_plus:
 
@@ -67,8 +67,8 @@ A combined heat and power plant produces electricity, in this case from natural
                     carrier_out_2: heat
                     primary_carrier_out: electricity
                 constraints:
-                    energy_eff: 0.45
-                    energy_cap_max: 100
+                    flow_eff: 0.45
+                    flow_cap_max: 100
                     carrier_ratios.carrier_out_2.heat: 0.8
 
 
@@ -89,8 +89,8 @@ The output energy from the heat pump can be *either* heat or cooling, simulating
             primary_carrier_out: heat
 
         constraints:
-            energy_eff: 1
-            energy_cap_max: 100
+            flow_eff: 1
+            flow_cap_max: 100
             carrier_ratios:
                 carrier_out:
                     heat: 5
@@ -124,14 +124,14 @@ A CCHP plant can use generated heat to produce cooling via an absorption chiller
                     primary_carrier_out: electricity
 
                 constraints:
-                    energy_eff: 0.45
-                    energy_cap_max: 100
+                    flow_eff: 0.45
+                    flow_cap_max: 100
                     carrier_ratios.carrier_out_2: {heat: 0.8, cooling: 0.5}
 
 Advanced gas turbine
 --------------------
 
-This technology can choose to burn methane (CH:sub:`4`) or send hydrogen (H:sub:`2`) through a fuel cell to produce electricity. One unit of carrier_in can be met by any combination of methane and hydrogen. If all methane, 0.5 units of carrier_out would be produced for 1 unit of carrier_in (energy_eff). If all hydrogen, 0.25 units of carrier_out would be produced for the same amount of carrier_in (energy_eff * hydrogen carrier ratio).
+This technology can choose to burn methane (CH:sub:`4`) or send hydrogen (H:sub:`2`) through a fuel cell to produce electricity. One unit of carrier_in can be met by any combination of methane and hydrogen. If all methane, 0.5 units of carrier_out would be produced for 1 unit of carrier_in (flow_eff). If all hydrogen, 0.25 units of carrier_out would be produced for the same amount of carrier_in (flow_eff * hydrogen carrier ratio).
 
 .. figure:: images/conversion_plus_gas.*
 
@@ -144,8 +144,8 @@ This technology can choose to burn methane (CH:sub:`4`) or send hydrogen (H:sub:
             carrier_out: electricity
 
         constraints:
-            energy_eff: 0.5
-            energy_cap_max: 100
+            flow_eff: 0.5
+            flow_cap_max: 100
             carrier_ratios:
                 carrier_in: {methane: 1, hydrogen: 0.5}
 
@@ -175,8 +175,8 @@ There are few instances where using the full capacity of a conversion_plus tech
                     primary_carrier_out: electricity
 
                 constraints:
-                    energy_eff: 1
-                    energy_cap_max: 100
+                    flow_eff: 1
+                    flow_cap_max: 100
                     carrier_ratios:
                         carrier_in: {coal: 1.2, gas: 1, oil: 1.6}
                         carrier_in_2: {biomass: 1, waste: 1.25}
@@ -186,28 +186,28 @@ There are few instances where using the full capacity of a conversion_plus tech
 
 A ``primary_carrier_out`` must be defined when there are multiple ``carrier_out`` values defined, similarly ``primary_carrier_in`` can be defined for ``carrier_in``. `primary_carriers` can be defined as any carrier in a technology's input/output carriers (including secondary and tertiary carriers). The chosen output carrier will be the one to which production costs are applied (reciprocally, input carrier for consumption costs).
 
-.. note:: ``Conversion_plus`` technologies can also export any one of their output carriers, by specifying that carrier as ``carrier_export``.
+.. note:: ``Conversion_plus`` technologies can also export any one of their output carriers, by specifying that carrier as the ``export_carrier``.
 
--------------------------
-Resource area constraints
--------------------------
+--------------------
+Area use constraints
+--------------------
 
 Several optional constraints can be used to specify area-related restrictions on technology use.
 
-To make use of these constraints, one should set ``resource_unit: energy_per_area`` for the given technologies. This scales the available resource at a given location for a given technology with its ``resource_area`` decision variable.
+To make use of these constraints, one should set ``source_unit: per_area`` for the given technologies. This scales the available source at a given location for a given technology with its ``area_use`` decision variable.
 
 The following related settings are available:
 
-* ``resource_area_equals``, ``resource_area_max``, ``resource_area_min``: Set uppper or lower bounds on resource_area or force it to a specific value
-* ``resource_area_per_energy_cap``: False by default, but if set to true, it forces ``resource_area`` to follow ``energy_cap`` with the given numerical ratio (e.g. setting to 1.5 means that ``resource_area == 1.5 * energy_cap``)
+* ``area_use_equals``, ``area_use_max``, ``area_use_min``: Set uppper or lower bounds on area_use or force it to a specific value
+* ``area_use_per_flow_cap``: False by default, but if set to true, it forces ``area_use`` to follow ``flow_cap`` with the given numerical ratio (e.g. setting to 1.5 means that ``area_use == 1.5 * flow_cap``)
 
-By default, ``resource_area_max`` is infinite and ``resource_area_min`` is 0 (zero).
+By default, ``area_use_max`` is infinite and ``area_use_min`` is 0 (zero).
 
 ----------------------------------
 Per-distance constraints and costs
 ----------------------------------
 
-Transmission technologies can additionally specify per-distance efficiency (loss) with ``energy_eff_per_distance`` and per-distance costs with ``energy_cap_per_distance``:
+Transmission technologies can additionally specify per-distance efficiency (loss) with ``flow_eff_per_distance`` and per-distance costs with ``flow_cap_per_distance``:
 
 .. code-block:: yaml
 
@@ -217,11 +217,11 @@ Transmission technologies can additionally specify per-distance efficiency (loss
                 ...
             constraints:
                 # "efficiency" (1-loss) per unit of distance
-                energy_eff_per_distance: 0.99
+                flow_eff_per_distance: 0.99
             costs:
                 monetary:
                     # cost per unit of distance
-                    energy_cap_per_distance: 10
+                    flow_cap_per_distance: 10
 
 The distance is specified in transmission links:
 
@@ -232,7 +232,7 @@ The distance is specified in transmission links:
             my_transmission_tech:
                 distance: 500
                 constraints:
-                    energy_cap.max: 10000
+                    flow_cap.max: 10000
 
 If no distance is given, but the locations have been given lat and lon coordinates, Calliope will compute distances automatically (based on the length of a straight line connecting the locations).
 
@@ -270,7 +270,7 @@ Revenue and export
 
 It is possible to specify revenues for technologies simply by setting a negative cost value. For example, to consider a feed-in tariff for PV generation, it could be given a negative operational cost equal to the real operational cost minus the level of feed-in tariff received.
 
-Export is an extension of this, allowing an energy carrier to be removed from the system without meeting demand. This is analogous to e.g. domestic PV technologies being able to export excess electricity to the national grid. A cost (or negative cost: revenue) can then be applied to export.
+Export is an extension of this, allowing a carrier to be removed from the system without meeting demand. This is analogous to e.g. domestic PV technologies being able to export excess electricity to the national grid. A cost (or negative cost: revenue) can then be applied to export.
 
 .. note:: Negative costs can be applied to capacity costs, but the user must an ensure a capacity limit has been set. Otherwise, optimisation will be unbounded.
 
@@ -302,18 +302,18 @@ Possible ``group_share`` constraints with carrier-specific settings are:
 
 Possible ``group_share`` constraints with carrier-independent settings are:
 
-* ``energy_cap_min``
-* ``energy_cap_max``
-* ``energy_cap_equals``
+* ``flow_cap_min``
+* ``flow_cap_max``
+* ``flow_cap_equals``
 
-These can be implemented as, for example, to force at most 20% of ``energy_cap`` to come from the two listed technologies:
+These can be implemented as, for example, to force at most 20% of ``flow_cap`` to come from the two listed technologies:
 
 .. code-block:: yaml
 
     model:
         group_share:
             csp,cold_fusion:
-                energy_cap_max: 0.20
+                flow_cap_max: 0.20
 
 ------------------------------------
 Binary and mixed-integer constraints
@@ -331,10 +331,10 @@ By applying ``units.max``, ``units.min``, or ``units.equals`` to a technology, t
 
 .. seealso:: :ref:`milp_example_model`
 
-Asynchronous energy production/consumption
-------------------------------------------
+Asynchronous flow in/out
+------------------------
 
-The ``asynchronous_prod_con`` binary constraint ensures that only one of ``carrier_prod`` and ``carrier_con`` can be non-zero in a given timestep.
+The ``async_flow_switch`` binary variable ensures that only one of ``flow_out`` and ``flow_in`` can be non-zero in a given timestep.
 
 This constraint can be applied to storage or transmission technologies. This example shows use with a heat transmission technology:
 
@@ -344,7 +344,7 @@ This constraint can be applied to storage or transmission technologies. This exa
    :start-after: # heat_pipes-start
    :end-before: # heat_pipes-end
 
-In the above example, heat pipes which distribute thermal energy in the network may be prone to dissipating heat in an unphysical way. I.e. given that they have distribution losses associated with them, in any given timestep, a link could produce and consume energy in the same timestep, losing energy to the atmosphere in both instances, but having a net energy transmission of zero. This might allow e.g. a CHP facility to overproduce heat to produce more cheap electricity, and have some way of dumping that heat. Enabling the ``asynchronous_prod_con`` constraint ensures that this does not happen.
+In the above example, heat pipes which distribute thermal energy in the network may be prone to dissipating heat in an unphysical way. I.e. given that they have distribution losses associated with them, in any given timestep, a link could produce and consume energy in the same timestep, losing energy to the atmosphere in both instances, but having a net energy transmission of zero. This might allow e.g. a CHP facility to overproduce heat to produce more cheap electricity, and have some way of dumping that heat. Enabling the ``force_async_flow`` parameter ensures that this does not happen.
 
 -------------------------------
 User-defined custom constraints