Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lossy Bidirectional Links #1192

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
143 changes: 67 additions & 76 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,17 +190,17 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<tbody>
<tr>
<td align="center">
<a href="https://github.com/FabianHofmann">
<img src="https://avatars.githubusercontent.com/u/19226431?v=4" width="100;" alt="FabianHofmann"/>
<a href="https://github.com/Eric-Nitschke">
<img src="https://avatars.githubusercontent.com/u/152230633?v=4" width="100;" alt="Eric-Nitschke"/>
<br />
<sub><b>Fabian Hofmann</b></sub>
<sub><b>Eric-Nitschke</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/fneum">
<img src="https://avatars.githubusercontent.com/u/29101152?v=4" width="100;" alt="fneum"/>
<a href="https://github.com/davide-f">
<img src="https://avatars.githubusercontent.com/u/67809479?v=4" width="100;" alt="davide-f"/>
<br />
<sub><b>Fabian Neumann</b></sub>
<sub><b>Davide-f</b></sub>
</a>
</td>
<td align="center">
Expand All @@ -210,87 +210,29 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<sub><b>Ekaterina</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/euronion">
<img src="https://avatars.githubusercontent.com/u/42553970?v=4" width="100;" alt="euronion"/>
<br />
<sub><b>Euronion</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/Justus-coded">
<img src="https://avatars.githubusercontent.com/u/44394641?v=4" width="100;" alt="Justus-coded"/>
<br />
<sub><b>Justus Ilemobayo</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/mnm-matin">
<img src="https://avatars.githubusercontent.com/u/45293386?v=4" width="100;" alt="mnm-matin"/>
Comment on lines -213 to -229
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess those are changes from an automated run of github workflow? 😄 Don't think we need them in the PR. Could you please remove them?

The recent PR #1210 changes a mode of the automated updates of the contributors list to a pre-scheduled one. So we don't loose track of contributions but PRs won't have those irrelevant additions anymore.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree.

<br />
<sub><b>Mnm-matin</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/martacki">
<img src="https://avatars.githubusercontent.com/u/53824825?v=4" width="100;" alt="martacki"/>
<br />
<sub><b>Martha Frysztacki</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/LukasFrankenQ">
<img src="https://avatars.githubusercontent.com/u/55196140?v=4" width="100;" alt="LukasFrankenQ"/>
<br />
<sub><b>Lukas Franken</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/pz-max">
<img src="https://avatars.githubusercontent.com/u/61968949?v=4" width="100;" alt="pz-max"/>
<br />
<sub><b>Max Parzen</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/davide-f">
<img src="https://avatars.githubusercontent.com/u/67809479?v=4" width="100;" alt="davide-f"/>
<br />
<sub><b>Davide-f</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/koen-vg">
<img src="https://avatars.githubusercontent.com/u/74298901?v=4" width="100;" alt="koen-vg"/>
<br />
<sub><b>Koen Van Greevenbroek</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/hazemakhalek">
<img src="https://avatars.githubusercontent.com/u/87850910?v=4" width="100;" alt="hazemakhalek"/>
<br />
<sub><b>Hazem</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/energyLS">
<img src="https://avatars.githubusercontent.com/u/89515385?v=4" width="100;" alt="energyLS"/>
<br />
<sub><b>EnergyLS</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/AnasAlgarei">
<img src="https://avatars.githubusercontent.com/u/101210563?v=4" width="100;" alt="AnasAlgarei"/>
<br />
<sub><b>AnasAlgarei</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/yerbol-akhmetov">
<img src="https://avatars.githubusercontent.com/u/113768325?v=4" width="100;" alt="yerbol-akhmetov"/>
Expand All @@ -305,13 +247,27 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<sub><b>DeniseGiub</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/FabianHofmann">
<img src="https://avatars.githubusercontent.com/u/19226431?v=4" width="100;" alt="FabianHofmann"/>
<br />
<sub><b>Fabian Hofmann</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/GbotemiB">
<img src="https://avatars.githubusercontent.com/u/48842684?v=4" width="100;" alt="GbotemiB"/>
<br />
<sub><b>Emmanuel Bolarinwa</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/mnm-matin">
<img src="https://avatars.githubusercontent.com/u/45293386?v=4" width="100;" alt="mnm-matin"/>
<br />
<sub><b>Mnm-matin</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/Eddy-JV">
<img src="https://avatars.githubusercontent.com/u/75539255?v=4" width="100;" alt="Eddy-JV"/>
Expand Down Expand Up @@ -349,22 +305,29 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<sub><b>GridGrapher</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/martacki">
<img src="https://avatars.githubusercontent.com/u/53824825?v=4" width="100;" alt="martacki"/>
<br />
<sub><b>Martha Frysztacki</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/glenkiely-ieg">
<img src="https://avatars.githubusercontent.com/u/99269783?v=4" width="100;" alt="glenkiely-ieg"/>
<br />
<sub><b>Null</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/cpschau">
<img src="https://avatars.githubusercontent.com/u/124347782?v=4" width="100;" alt="cpschau"/>
<br />
<sub><b>Cschau</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/Emre-Yorat89">
<img src="https://avatars.githubusercontent.com/u/62134151?v=4" width="100;" alt="Emre-Yorat89"/>
Expand Down Expand Up @@ -393,22 +356,43 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<sub><b>Ekaterina-Vo</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/euronion">
<img src="https://avatars.githubusercontent.com/u/42553970?v=4" width="100;" alt="euronion"/>
<br />
<sub><b>Euronion</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/lkstrp">
<img src="https://avatars.githubusercontent.com/u/62255395?v=4" width="100;" alt="lkstrp"/>
<br />
<sub><b>Lukas Trippe</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/LukasFrankenQ">
<img src="https://avatars.githubusercontent.com/u/55196140?v=4" width="100;" alt="LukasFrankenQ"/>
<br />
<sub><b>Lukas Franken</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/AnasAlgarei">
<img src="https://avatars.githubusercontent.com/u/101210563?v=4" width="100;" alt="AnasAlgarei"/>
<br />
<sub><b>AnasAlgarei</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/Tooblippe">
<img src="https://avatars.githubusercontent.com/u/805313?v=4" width="100;" alt="Tooblippe"/>
<br />
<sub><b>Tobias</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/asolavi">
<img src="https://avatars.githubusercontent.com/u/131155817?v=4" width="100;" alt="asolavi"/>
Expand All @@ -422,6 +406,15 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<br />
<sub><b>Null</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/koen-vg">
<img src="https://avatars.githubusercontent.com/u/74298901?v=4" width="100;" alt="koen-vg"/>
<br />
<sub><b>Koen Van Greevenbroek</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/danielelerede-oet">
Expand Down Expand Up @@ -451,15 +444,15 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<sub><b>Ryan</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/ollie-bell">
<img src="https://avatars.githubusercontent.com/u/56110893?v=4" width="100;" alt="ollie-bell"/>
<br />
<sub><b>Null</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/juli-a-ko">
<img src="https://avatars.githubusercontent.com/u/126512394?v=4" width="100;" alt="juli-a-ko"/>
Expand Down Expand Up @@ -495,15 +488,15 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<sub><b>Null</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/kma33">
<img src="https://avatars.githubusercontent.com/u/25573938?v=4" width="100;" alt="kma33"/>
<br />
<sub><b>Katherine M. Antonio</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/jessLryan">
<img src="https://avatars.githubusercontent.com/u/122939887?v=4" width="100;" alt="jessLryan"/>
Expand Down Expand Up @@ -539,8 +532,6 @@ The documentation is available here: [documentation](https://pypsa-earth.readthe
<sub><b>André Cristóvão Neves Ferreira</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/AlexanderMeisinger">
<img src="https://avatars.githubusercontent.com/u/91368938?v=4" width="100;" alt="AlexanderMeisinger"/>
Expand Down
2 changes: 2 additions & 0 deletions doc/release_notes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ E.g. if a new rule becomes available describe how to use it `make test` and in o

* Adds CI to update keep pinned environment files up to date. `PR #1183 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1183>`__

* Fix lossy bidirectional links, especially H2 pipelines, which would sometimes gain H2 instead of losing it. `PR #1192 <https://github.com/pypsa-meets-earth/pypsa-earth/pull/1192>`__

ekatef marked this conversation as resolved.
Show resolved Hide resolved
PyPSA-Earth 0.4.1
=================

Expand Down
30 changes: 30 additions & 0 deletions scripts/add_extra_components.py
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,36 @@ def attach_hydrogen_pipelines(n, costs, config):
carrier="H2 pipeline",
Copy link
Collaborator

@Eddy-JV Eddy-JV Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

efficiency=costs.at["H2 pipeline", "efficiency"], to be removed as it is set now in lossy_bidirectional_links as per below comments.

)

# setup pipelines as bidirectional and lossy
lossy_bidirectional_links(n, "H2 pipeline")
Copy link
Collaborator

@Eddy-JV Eddy-JV Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on all my suggestions, this would be change to:

    # Add option to specify losses for bidirectional links, e.g. pipelines or HVDC links, 
    # in configuration file under sector: transmission_efficiency:. 
    # Users can specify static or length-dependent values as well as a length-dependent electricity demand for compression,
    #  which is implemented as a multi-link to the local electricity buses. 
    # The bidirectional links will then be split into two unidirectional links with linked capacities

    for k, v in snakemake.params.transmission_efficiency.items():
        if k == "H2 pipeline":
            lossy_bidirectional_links(n, k, v)

And in the snakefile please add to the rule add_extra_components:

       params:
              transmission_efficiency=config["sector"]["transmission_efficiency"],



def lossy_bidirectional_links(n: pypsa.components.Network, carrier: str):
Copy link
Collaborator

@Eddy-JV Eddy-JV Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it is better to move lossy_bidirectional_links to helpers.py, beacuse it will be used more than once throughout the workflow.

Additionally, I would prefer to apply directly the length based pipeline efficiency that I saw in PR #1215. Consequently changing the function to the following similarly to PyPSA-Eur but without the .location at the moment:

  def lossy_bidirectional_links(n, carrier, efficiencies={}):
  
      ``` Split bidirectional links into two unidirectional links to include transmission losses. ```
      
      carrier_i = n.links.query("carrier == @carrier").index
  
      if (
          not any((v != 1.0) or (v >= 0) for v in efficiencies.values())
          or carrier_i.empty
      ):
          return
  
      efficiency_static = efficiencies.get("efficiency_static", 1)
      efficiency_per_1000km = efficiencies.get("efficiency_per_1000km", 1)
      compression_per_1000km = efficiencies.get("compression_per_1000km", 0)
  
      logger.info(
          f"Specified losses for {carrier} transmission "
          f"(static: {efficiency_static}, per 1000km: {efficiency_per_1000km}, compression per 1000km: {compression_per_1000km}). "
          "Splitting bidirectional links."
      )
  
      n.links.loc[carrier_i, "p_min_pu"] = 0
      n.links.loc[carrier_i, "efficiency"] = (
          efficiency_static
          * efficiency_per_1000km ** (n.links.loc[carrier_i, "length"] / 1e3)
      )
      rev_links = (
          n.links.loc[carrier_i].copy().rename({"bus0": "bus1", "bus1": "bus0"}, axis=1)
      )
      rev_links["length_original"] = rev_links["length"]
      rev_links["capital_cost"] = 0
      rev_links["length"] = 0
      rev_links["reversed"] = True
      rev_links.index = rev_links.index.map(lambda x: x + "-reversed")
  
      n.links = pd.concat([n.links, rev_links], sort=False)
      n.links["reversed"] = n.links["reversed"].fillna(False).infer_objects(copy=False)
      n.links["length_original"] = n.links["length_original"].fillna(n.links.length)
  
      # do compression losses after concatenation to take electricity consumption at bus0 in either direction
      carrier_i = n.links.query("carrier == @carrier").index
      if compression_per_1000km > 0:
          n.links.loc[carrier_i, "bus2"] = n.links.loc[
              carrier_i, "bus0"
          ].str.removesuffix(' ')
          # TODO: use these lines to set bus 2 instead, once n.buses.location is functional and remove bus_suffix.
          """
          n.links.loc[carrier_i, "bus2"] = n.links.loc[carrier_i, "bus0"].map(
              n.buses.location
          )  # electricity
          """
          n.links.loc[carrier_i, "efficiency2"] = (
              -compression_per_1000km * n.links.loc[carrier_i, "length_original"] / 1e3
          )

As for the config file, the following change is advised in sector :

  transmission_efficiency:
    H2 pipeline:
      efficiency_per_1000km: 0.983 # DEA technology data. Energy losses, lines 5000-20000 MW, [%/1000 km] and year 2050
      compression_per_1000km: 0.015 # DEA technology data. year 2050

If this is done as above, then PR #1215 is not needed anymore and issue #1213 can be closed.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea, to move the function to helpers!

I think it's nice to implement both PR's together, but I'd prefer to leave the functions seperate (eventhough PyPSA-Eur combines them) to make them more reusable. I think it's quite plausible, that some new components might be bidirectional and lossy, but without a length-based efficiency, or have a length-based efficiency, but are only unidirectional.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Eddy-JV regarding the efficiency_per_1000km you provided:
Is it intentional, that the pipeline loses hydrogen substance (which results from an efficiency_per_1000km < 1) or should it just require more electricity per distance, which is depicted by compression_per_1000km.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Eric-Nitschke Thank you for catching that. In the Hydrogen case the efficiency should be efficiency_per_1000km: 1 .
In DEA they consider the energy needs for compressors within the efficiency per 1000km for different classes of pipeline sizes. So I considered the mean of the largest two classes: 5000-20000 MW and >20000 MW for 2020, 2030 and 2050. Hence, the config will look as follows:

  transmission_efficiency:
    H2 pipeline:
      efficiency_per_1000km: 1 
      compression_per_1000km: 0.017 # DEA technology data. Mean of  Energy losses, lines 5000-20000 MW and lines >20000 MW for 2020, 2030 and 2050, [%/1000 km]


"Split bidirectional links into two unidirectional links to include transmission losses."
Comment on lines +273 to +274
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually, we are using """ to write docstrings


carrier_i = n.links.query("carrier == @carrier").index

if carrier_i.empty:
return

logger.info(f"Splitting bidirectional links with the carrier {carrier}")

n.links.loc[carrier_i, "p_min_pu"] = 0

rev_links = (
n.links.loc[carrier_i].copy().rename({"bus0": "bus1", "bus1": "bus0"}, axis=1)
)
rev_links["length_original"] = rev_links["length"]
rev_links["capital_cost"] = 0
rev_links["length"] = 0
rev_links["reversed"] = True
rev_links.index = rev_links.index.map(lambda x: x + "-reversed")

n.links = pd.concat([n.links, rev_links], sort=False)
n.links["reversed"] = n.links["reversed"].fillna(False).infer_objects(copy=False)
n.links["length_original"] = n.links["length_original"].fillna(n.links.length)


if __name__ == "__main__":
if "snakemake" not in globals():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please accomodate for override components by:

Removing:
n = pypsa.Network(snakemake.input.network)

Adding:
overrides = override_component_attrs(snakemake.input.overrides)
n = pypsa.Network(snakemake.input.network, override_component_attrs=overrides)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davide-f @ekatef please check if it is ok to override components in the electricity only model.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Eddy-JV, frankly speaking I'm not quite aware of the proper approach to overwrite attributes. I assume the need arises from the need to translate a power-only model into the sector-coupled version, but having a bit more details would be appreciated 🙂

Copy link
Author

@Eric-Nitschke Eric-Nitschke Dec 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @ekatef, in this case the need arises from adding a compression electricity demand to the pipelines. This requires a third bus (bus2) for the link and a corresponding efficiency (efficiency2). The override_component_attr function adds these fields (and more) to the network.

Expand Down
38 changes: 38 additions & 0 deletions scripts/solve_network.py
Original file line number Diff line number Diff line change
Expand Up @@ -854,6 +854,43 @@ def add_existing(n):
n.generators.loc[tech_index, tech] = existing_res


def add_lossy_bidirectional_link_constraints(n: pypsa.components.Network) -> None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did this slightly different (See below). Feel free to check and comment if you do not agree:

def add_lossy_bidirectional_link_constraints(n):
    """
        Ensures that the two links simulating a bidirectional_link are extended the same amount.
    """
    if not n.links.p_nom_extendable.any() or "reversed" not in n.links.columns:
        return

    # Ensure 'reversed' column is boolean
    n.links["reversed"] = n.links.reversed.fillna(0).astype(bool)
    carriers = n.links.loc[n.links.reversed, "carrier"].unique()

    # Get forward link indices (non-reversed)
    forward_i = n.links.query(
        "carrier in @carriers and ~reversed and p_nom_extendable"
    ).index

    # Function to get backward (reversed) indices corresponding to forward links
    def get_backward_i(forward_i):
        return pd.Index(
            [
                re.sub(r"-(\d{4})$", r"-reversed-\1", s)
                if re.search(r"-\d{4}$", s)
                else s + "-reversed"
                for s in forward_i
            ]
        )

    backward_i = get_backward_i(forward_i)

    # Get the p_nom optimization variables for the links using the get_var function
    links_p_nom = get_var(n, "Link", "p_nom")

    # Only consider forward and backward links that are present in the optimization variables
    subset_forward = forward_i.intersection(links_p_nom.index)
    subset_backward = backward_i.intersection(links_p_nom.index)

    # Ensure we have a matching number of forward and backward links
    if len(subset_forward) != len(subset_backward):
        raise ValueError("Mismatch between forward and backward links.")

    # For each forward index, find the corresponding backward index and define the constraint
    for fwd, bwd in zip(subset_forward, subset_backward):
        lhs = linexpr((1, links_p_nom.loc[bwd]))  # LHS (backward link)
        rhs = linexpr((-1, links_p_nom.loc[fwd]))  # RHS (forward link with -1 coefficient)

        # Define the constraint for this pair of forward-backward link
        define_constraints(
            n,
            lhs + rhs,  # LHS - RHS = 0 (equality constraint)
            "=",
            0,  # Right-hand side is 0 because we want them to be equal
            name="Link",
            attr=f"bidirectional_sync_{fwd}",  # Custom name for this constraint (per pair)
            axes=[pd.Index([fwd])]  # The axes refer to the forward link
        )

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Eddy-JV, even though the additional checks you proposed here are not neccessary right now, I think it's great to future-proof the code right now. I'm going to include the additional checks in the final version.
However, I wouldn't add the constraints one by one for each component, since no other constraint is added that way.

"""
Comment on lines +857 to +858
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Types hinting is generally a great idea, but we don't use. I'd suggest to remove it here for consistency.

Happy to discuss an overall revision to implement type hinting across the whole codebase if feels handy

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. Maybe we can put this as a seperate issue if needed.

Ensures that the two links simulating a bidirectional_link are extended the same amount.
"""

if not n.links.p_nom_extendable.any() or "reversed" not in n.links.columns:
return

n.links["reversed"] = n.links.reversed.fillna(0).astype(bool)
carriers = n.links.loc[n.links.reversed, "carrier"].unique() # noqa: F841
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that noqa: F841 follows PyPSA-Eur implementation. Haven't you tried to understand what is the reason of it's inclusion into the code?

Looking into flake8 error codes, it appears that # noqa: F841 suppresses warnings generated when local variable name is assigned to but never used. Frankly speaking, I'm not even sure it's working in our case as flake seems to be not included into our environment. Can we probably just remove it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Short answer: carriers is used in the lines just below, but the interpreter doesn't realize it, because its used in with .query(). Thats why noqa: F841 is used.

I'll write a long answer tomorrow.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slightly longer answer:

forward_i = n.links.query(
        "carrier in @carriers and ~reversed and p_nom_extendable"
    ).index

uses carriers, but because pandas.DataFrame.query() takes a string as an input, the python interpreter doesn't realize that carriers is used via the "@carriers".

I don't think it is necessary to use .query() here. Instead, we could use .loc[]:

forward_i = n.links.loc[
        (n.links.carrier in carriers) and
        ~n.links.reversed and
        n.links.p_nom_extendable
].index

In general, I think we should discuss whether we should phase out the use of .query() and replace it with .loc[] for all of PyPSA-Earth to make it more consistent and readable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, ok! Thanks a lot for investigating this! Agree with the proposal on replacing here query with loc

Regarding the overall code refactoring, totally agree that there is a room for improvement and you are very welcome to contribute there. Frankly speaking, I'm not ready to confirm that query is less readable than loc for all the cases. But you may be right that it could be over-used for sure, and happy to discuss you propositions!


forward_i = n.links.query(
"carrier in @carriers and ~reversed and p_nom_extendable"
).index

def get_backward_i(forward_i):
return pd.Index(
[
(
re.sub(r"-(\d{4})$", r"-reversed-\1", s)
if re.search(r"-\d{4}$", s)
else s + "-reversed"
)
for s in forward_i
]
)
Comment on lines +872 to +882
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for better understanding, what is a particular reason to apply here regular expressions? Again, I understand that implementation follows PyPSA-Eur and I'm perfectly fine with leaving it there. Just trying to draft some approach to further improvements which we could probably add as TODO if it would make sense 🙂

I guess sub is used to extract the identifications from the links indices and build corresponding names for the reversed links. However, regexpr are not exactly readable, and the implementation can easily break if PyPSA conventions would change. I wonder if some pandas string functions can provide a good replacement?

Regular expressions are

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regular expression enables it us to easily change indices tha end in a four digit number to not end in reversed but have it in the middle instead. I.e. "H2 pipeline DZ0 0-DZ0-1234" to "H2 pipeline DZ0 0-DZ0-reversed-1234" instead of "H2 pipeline DZ0 0-DZ0-1234-reversed".
That being said, no component in PyPSA-Earth ends in a four digit number and the H2 pipelines definitely don't. I'm not even sure if PyPSA-Eur does anymore, since it never looks for a r"-(\d{4})$" ending.

If we assume that the four digit ending can be ignored, the whole thing could be replaced with something like backward_i = forward_i + "-reversed" or backward_i = forward_i.apply(lambda x: f"{x}-reversed").

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the explanations. Do you see any advantages in having "reversed" in the middle of the link name?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ekatef I don't see any advantage to it.


backward_i = get_backward_i(forward_i)

lhs = linexpr(
(1, get_var(n, "Link", "p_nom")[backward_i].to_numpy()),
(-1, get_var(n, "Link", "p_nom")[forward_i].to_numpy()),
)

define_constraints(n, lhs, "=", 0, "Link-bidirectional_sync")


def extra_functionality(n, snapshots):
"""
Collects supplementary constraints which will be passed to
Expand Down Expand Up @@ -881,6 +918,7 @@ def extra_functionality(n, snapshots):
if "EQ" in o:
add_EQ_constraints(n, o)
add_battery_constraints(n)
add_lossy_bidirectional_link_constraints(n)

if (
snakemake.config["policy_config"]["hydrogen"]["temporal_matching"]
Expand Down