Lossy Bidirectional Links #1192
Conversation
Summary: - fixed an existing bug regarding bidirectional lossy links that affected hydrogen pipelines. - the fix primarily utilizes code already in use by PyPSA-Eur. Changes: - added a function to scripts/add_extra_components.py that "splits" bidirectional links into two unidirectional links. This is based upon the similar PyPSA-Eur implementation, which is slightly more advanced to accommodate PyPSA-Eurs more advanced efficiency calculations. - added a function to scripts/solve_network.py that adds an additional constraint to the network, which ensures that both unidirectional links share the same nominal power. This is a modified version of the PyPSA-Eur function, as PyPSA-Eur uses linopy while the base PyPSA-Earth version operates on lopf. Reason: - without this commit, lossy bidirectional links are able to "generate" power by operating at a negative power level. - this stems from the general link implementation, which applies p1 = -eta*p0 to links, regardless whether power flows from bus0 to bus1 or from bus1 to bus0. - the current hydrogen pipeline implementation adds them to the network as lossy bidirectional links and is thus able to "generate" hydrogen. - this might affect other lossy bidirectional components added by the sector network as well.
for more information, see https://pre-commit.ci
|
Many thanks @Eric-Nitschke :D That's a quite neat application! |
Changes: - add a describtion of this pull request to the release_notes.rst
|
@davide-f Thanks :) |
- correct spelling mistakes in doc/release_notes.rst
There was a problem hiding this comment.
Hello @Eric-Nitschke, thanks a lot for investigating the issue and contributing the fix!
Have added some technical comments mainly aimed at better understanding and maintainability improvements. I recognise that the PR follows PyPSA-Eur implementation which is more or less state-of-the-art, but it would be definitely great to have deeper understanding of possible further improvements on some weak points. Your insights in this regard would be very helpful.
Personally, I think that this PR can be probably merged as-is as it feels like a much needed bug-fix. It would be great to have a review of @Eddy-JV who has been working on implementing similar solutions in PyPSA-Eur and definitely has a better overview
| <td align="center"> | ||
| <a href="https://github.com/euronion"> | ||
| <img src="https://avatars.githubusercontent.com/u/42553970?v=4" width="100;" alt="euronion"/> | ||
| <br /> | ||
| <sub><b>Euronion</b></sub> | ||
| </a> | ||
| </td> | ||
| <td align="center"> | ||
| <a href="https://github.com/Justus-coded"> | ||
| <img src="https://avatars.githubusercontent.com/u/44394641?v=4" width="100;" alt="Justus-coded"/> | ||
| <br /> | ||
| <sub><b>Justus Ilemobayo</b></sub> | ||
| </a> | ||
| </td> | ||
| <td align="center"> | ||
| <a href="https://github.com/mnm-matin"> | ||
| <img src="https://avatars.githubusercontent.com/u/45293386?v=4" width="100;" alt="mnm-matin"/> |
There was a problem hiding this comment.
I guess those are changes from an automated run of github workflow? 😄 Don't think we need them in the PR. Could you please remove them?
The recent PR #1210 changes a mode of the automated updates of the contributors list to a pre-scheduled one. So we don't loose track of contributions but PRs won't have those irrelevant additions anymore.
|
|
||
| "Split bidirectional links into two unidirectional links to include transmission losses." |
There was a problem hiding this comment.
Usually, we are using """ to write docstrings
| def add_lossy_bidirectional_link_constraints(n: pypsa.components.Network) -> None: | ||
| """ |
There was a problem hiding this comment.
Types hinting is generally a great idea, but we don't use. I'd suggest to remove it here for consistency.
Happy to discuss an overall revision to implement type hinting across the whole codebase if feels handy
There was a problem hiding this comment.
Agree. Maybe we can put this as a seperate issue if needed.
| return | ||
|
|
||
| n.links["reversed"] = n.links.reversed.fillna(0).astype(bool) | ||
| carriers = n.links.loc[n.links.reversed, "carrier"].unique() # noqa: F841 |
There was a problem hiding this comment.
I understand that noqa: F841 follows PyPSA-Eur implementation. Haven't you tried to understand what is the reason of it's inclusion into the code?
Looking into flake8 error codes, it appears that # noqa: F841 suppresses warnings generated when local variable name is assigned to but never used. Frankly speaking, I'm not even sure it's working in our case as flake seems to be not included into our environment. Can we probably just remove it?
There was a problem hiding this comment.
Short answer: carriers is used in the lines just below, but the interpreter doesn't realize it, because its used in with .query(). Thats why noqa: F841 is used.
I'll write a long answer tomorrow.
There was a problem hiding this comment.
Slightly longer answer:
forward_i = n.links.query(
"carrier in @carriers and ~reversed and p_nom_extendable"
).index
uses carriers, but because pandas.DataFrame.query() takes a string as an input, the python interpreter doesn't realize that carriers is used via the "@carriers".
I don't think it is necessary to use .query() here. Instead, we could use .loc[]:
forward_i = n.links.loc[
(n.links.carrier in carriers) and
~n.links.reversed and
n.links.p_nom_extendable
].index
In general, I think we should discuss whether we should phase out the use of .query() and replace it with .loc[] for all of PyPSA-Earth to make it more consistent and readable.
There was a problem hiding this comment.
Ahh, ok! Thanks a lot for investigating this! Agree with the proposal on replacing here query with loc
Regarding the overall code refactoring, totally agree that there is a room for improvement and you are very welcome to contribute there. Frankly speaking, I'm not ready to confirm that query is less readable than loc for all the cases. But you may be right that it could be over-used for sure, and happy to discuss you propositions!
| def get_backward_i(forward_i): | ||
| return pd.Index( | ||
| [ | ||
| ( | ||
| re.sub(r"-(\d{4})$", r"-reversed-\1", s) | ||
| if re.search(r"-\d{4}$", s) | ||
| else s + "-reversed" | ||
| ) | ||
| for s in forward_i | ||
| ] | ||
| ) |
There was a problem hiding this comment.
Just for better understanding, what is a particular reason to apply here regular expressions? Again, I understand that implementation follows PyPSA-Eur and I'm perfectly fine with leaving it there. Just trying to draft some approach to further improvements which we could probably add as TODO if it would make sense 🙂
I guess sub is used to extract the identifications from the links indices and build corresponding names for the reversed links. However, regexpr are not exactly readable, and the implementation can easily break if PyPSA conventions would change. I wonder if some pandas string functions can provide a good replacement?
Regular expressions are
There was a problem hiding this comment.
Regular expression enables it us to easily change indices tha end in a four digit number to not end in reversed but have it in the middle instead. I.e. "H2 pipeline DZ0 0-DZ0-1234" to "H2 pipeline DZ0 0-DZ0-reversed-1234" instead of "H2 pipeline DZ0 0-DZ0-1234-reversed".
That being said, no component in PyPSA-Earth ends in a four digit number and the H2 pipelines definitely don't. I'm not even sure if PyPSA-Eur does anymore, since it never looks for a r"-(\d{4})$" ending.
If we assume that the four digit ending can be ignored, the whole thing could be replaced with something like backward_i = forward_i + "-reversed" or backward_i = forward_i.apply(lambda x: f"{x}-reversed").
There was a problem hiding this comment.
Thanks a lot for the explanations. Do you see any advantages in having "reversed" in the middle of the link name?
There was a problem hiding this comment.
@ekatef I don't see any advantage to it.
There was a problem hiding this comment.
Thank you @Eric-Nitschke for this very important contribution.
I took the liberty to add some comments in order to cover this PR #1192 as well as PR #1215 and Issue #1213 at once. Feel free to discuss, if you think things can be done differently. Thanks again for the great work.
| <td align="center"> | ||
| <a href="https://github.com/euronion"> | ||
| <img src="https://avatars.githubusercontent.com/u/42553970?v=4" width="100;" alt="euronion"/> | ||
| <br /> | ||
| <sub><b>Euronion</b></sub> | ||
| </a> | ||
| </td> | ||
| <td align="center"> | ||
| <a href="https://github.com/Justus-coded"> | ||
| <img src="https://avatars.githubusercontent.com/u/44394641?v=4" width="100;" alt="Justus-coded"/> | ||
| <br /> | ||
| <sub><b>Justus Ilemobayo</b></sub> | ||
| </a> | ||
| </td> | ||
| <td align="center"> | ||
| <a href="https://github.com/mnm-matin"> | ||
| <img src="https://avatars.githubusercontent.com/u/45293386?v=4" width="100;" alt="mnm-matin"/> |
| lossy_bidirectional_links(n, "H2 pipeline") | ||
|
|
||
|
|
||
| def lossy_bidirectional_links(n: pypsa.components.Network, carrier: str): |
There was a problem hiding this comment.
I think that it is better to move lossy_bidirectional_links to helpers.py, beacuse it will be used more than once throughout the workflow.
Additionally, I would prefer to apply directly the length based pipeline efficiency that I saw in PR #1215. Consequently changing the function to the following similarly to PyPSA-Eur but without the .location at the moment:
def lossy_bidirectional_links(n, carrier, efficiencies={}):
``` Split bidirectional links into two unidirectional links to include transmission losses. ```
carrier_i = n.links.query("carrier == @carrier").index
if (
not any((v != 1.0) or (v >= 0) for v in efficiencies.values())
or carrier_i.empty
):
return
efficiency_static = efficiencies.get("efficiency_static", 1)
efficiency_per_1000km = efficiencies.get("efficiency_per_1000km", 1)
compression_per_1000km = efficiencies.get("compression_per_1000km", 0)
logger.info(
f"Specified losses for {carrier} transmission "
f"(static: {efficiency_static}, per 1000km: {efficiency_per_1000km}, compression per 1000km: {compression_per_1000km}). "
"Splitting bidirectional links."
)
n.links.loc[carrier_i, "p_min_pu"] = 0
n.links.loc[carrier_i, "efficiency"] = (
efficiency_static
* efficiency_per_1000km ** (n.links.loc[carrier_i, "length"] / 1e3)
)
rev_links = (
n.links.loc[carrier_i].copy().rename({"bus0": "bus1", "bus1": "bus0"}, axis=1)
)
rev_links["length_original"] = rev_links["length"]
rev_links["capital_cost"] = 0
rev_links["length"] = 0
rev_links["reversed"] = True
rev_links.index = rev_links.index.map(lambda x: x + "-reversed")
n.links = pd.concat([n.links, rev_links], sort=False)
n.links["reversed"] = n.links["reversed"].fillna(False).infer_objects(copy=False)
n.links["length_original"] = n.links["length_original"].fillna(n.links.length)
# do compression losses after concatenation to take electricity consumption at bus0 in either direction
carrier_i = n.links.query("carrier == @carrier").index
if compression_per_1000km > 0:
n.links.loc[carrier_i, "bus2"] = n.links.loc[
carrier_i, "bus0"
].str.removesuffix(' ')
# TODO: use these lines to set bus 2 instead, once n.buses.location is functional and remove bus_suffix.
"""
n.links.loc[carrier_i, "bus2"] = n.links.loc[carrier_i, "bus0"].map(
n.buses.location
) # electricity
"""
n.links.loc[carrier_i, "efficiency2"] = (
-compression_per_1000km * n.links.loc[carrier_i, "length_original"] / 1e3
)
As for the config file, the following change is advised in sector :
transmission_efficiency:
H2 pipeline:
efficiency_per_1000km: 0.983 # DEA technology data. Energy losses, lines 5000-20000 MW, [%/1000 km] and year 2050
compression_per_1000km: 0.015 # DEA technology data. year 2050
If this is done as above, then PR #1215 is not needed anymore and issue #1213 can be closed.
There was a problem hiding this comment.
Great idea, to move the function to helpers!
I think it's nice to implement both PR's together, but I'd prefer to leave the functions seperate (eventhough PyPSA-Eur combines them) to make them more reusable. I think it's quite plausible, that some new components might be bidirectional and lossy, but without a length-based efficiency, or have a length-based efficiency, but are only unidirectional.
There was a problem hiding this comment.
@Eddy-JV regarding the efficiency_per_1000km you provided:
Is it intentional, that the pipeline loses hydrogen substance (which results from an efficiency_per_1000km < 1) or should it just require more electricity per distance, which is depicted by compression_per_1000km.
There was a problem hiding this comment.
@Eric-Nitschke Thank you for catching that. In the Hydrogen case the efficiency should be efficiency_per_1000km: 1 .
In DEA they consider the energy needs for compressors within the efficiency per 1000km for different classes of pipeline sizes. So I considered the mean of the largest two classes: 5000-20000 MW and >20000 MW for 2020, 2030 and 2050. Hence, the config will look as follows:
transmission_efficiency:
H2 pipeline:
efficiency_per_1000km: 1
compression_per_1000km: 0.017 # DEA technology data. Mean of Energy losses, lines 5000-20000 MW and lines >20000 MW for 2020, 2030 and 2050, [%/1000 km]
| @@ -265,6 +265,36 @@ def attach_hydrogen_pipelines(n, costs, config): | |||
| carrier="H2 pipeline", | |||
There was a problem hiding this comment.
efficiency=costs.at["H2 pipeline", "efficiency"], to be removed as it is set now in lossy_bidirectional_links as per below comments.
| n.links = pd.concat([n.links, rev_links], sort=False) | ||
| n.links["reversed"] = n.links["reversed"].fillna(False).infer_objects(copy=False) | ||
| n.links["length_original"] = n.links["length_original"].fillna(n.links.length) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| if "snakemake" not in globals(): |
There was a problem hiding this comment.
Please accomodate for override components by:
Removing:
n = pypsa.Network(snakemake.input.network)
Adding:
overrides = override_component_attrs(snakemake.input.overrides)
n = pypsa.Network(snakemake.input.network, override_component_attrs=overrides)
There was a problem hiding this comment.
Hey @Eddy-JV, frankly speaking I'm not quite aware of the proper approach to overwrite attributes. I assume the need arises from the need to translate a power-only model into the sector-coupled version, but having a bit more details would be appreciated 🙂
There was a problem hiding this comment.
Hey @ekatef, in this case the need arises from adding a compression electricity demand to the pipelines. This requires a third bus (bus2) for the link and a corresponding efficiency (efficiency2). The override_component_attr function adds these fields (and more) to the network.
| @@ -265,6 +265,36 @@ def attach_hydrogen_pipelines(n, costs, config): | |||
| carrier="H2 pipeline", | |||
| ) | |||
|
|
|||
| # setup pipelines as bidirectional and lossy | |||
| lossy_bidirectional_links(n, "H2 pipeline") | |||
There was a problem hiding this comment.
Based on all my suggestions, this would be change to:
# Add option to specify losses for bidirectional links, e.g. pipelines or HVDC links,
# in configuration file under sector: transmission_efficiency:.
# Users can specify static or length-dependent values as well as a length-dependent electricity demand for compression,
# which is implemented as a multi-link to the local electricity buses.
# The bidirectional links will then be split into two unidirectional links with linked capacities
for k, v in snakemake.params.transmission_efficiency.items():
if k == "H2 pipeline":
lossy_bidirectional_links(n, k, v)
And in the snakefile please add to the rule add_extra_components:
params:
transmission_efficiency=config["sector"]["transmission_efficiency"],
| @@ -854,6 +854,43 @@ def add_existing(n): | |||
| n.generators.loc[tech_index, tech] = existing_res | |||
|
|
|||
|
|
|||
| def add_lossy_bidirectional_link_constraints(n: pypsa.components.Network) -> None: | |||
There was a problem hiding this comment.
I did this slightly different (See below). Feel free to check and comment if you do not agree:
def add_lossy_bidirectional_link_constraints(n):
"""
Ensures that the two links simulating a bidirectional_link are extended the same amount.
"""
if not n.links.p_nom_extendable.any() or "reversed" not in n.links.columns:
return
# Ensure 'reversed' column is boolean
n.links["reversed"] = n.links.reversed.fillna(0).astype(bool)
carriers = n.links.loc[n.links.reversed, "carrier"].unique()
# Get forward link indices (non-reversed)
forward_i = n.links.query(
"carrier in @carriers and ~reversed and p_nom_extendable"
).index
# Function to get backward (reversed) indices corresponding to forward links
def get_backward_i(forward_i):
return pd.Index(
[
re.sub(r"-(\d{4})$", r"-reversed-\1", s)
if re.search(r"-\d{4}$", s)
else s + "-reversed"
for s in forward_i
]
)
backward_i = get_backward_i(forward_i)
# Get the p_nom optimization variables for the links using the get_var function
links_p_nom = get_var(n, "Link", "p_nom")
# Only consider forward and backward links that are present in the optimization variables
subset_forward = forward_i.intersection(links_p_nom.index)
subset_backward = backward_i.intersection(links_p_nom.index)
# Ensure we have a matching number of forward and backward links
if len(subset_forward) != len(subset_backward):
raise ValueError("Mismatch between forward and backward links.")
# For each forward index, find the corresponding backward index and define the constraint
for fwd, bwd in zip(subset_forward, subset_backward):
lhs = linexpr((1, links_p_nom.loc[bwd])) # LHS (backward link)
rhs = linexpr((-1, links_p_nom.loc[fwd])) # RHS (forward link with -1 coefficient)
# Define the constraint for this pair of forward-backward link
define_constraints(
n,
lhs + rhs, # LHS - RHS = 0 (equality constraint)
"=",
0, # Right-hand side is 0 because we want them to be equal
name="Link",
attr=f"bidirectional_sync_{fwd}", # Custom name for this constraint (per pair)
axes=[pd.Index([fwd])] # The axes refer to the forward link
)
There was a problem hiding this comment.
Hey @Eddy-JV, even though the additional checks you proposed here are not neccessary right now, I think it's great to future-proof the code right now. I'm going to include the additional checks in the final version.
However, I wouldn't add the constraints one by one for each component, since no other constraint is added that way.
| def add_lossy_bidirectional_link_constraints(n: pypsa.components.Network) -> None: | ||
| """ |
There was a problem hiding this comment.
Agree. Maybe we can put this as a seperate issue if needed.
|
Hello @Eric-Nitschke and @Eddy-JV! Great work on reviewing the PR. @Eric-Nitschke do you need any support in revisions? If they look heavy, could it make sense to breakdown the task? That is a precious contribution, and it would be great to have it merged into the model 🙂 |
|
Hey @ekatef, in general I think the revisions look quite doable and I'm planning on finishing them before the holidays! Major changes:
Minor changes:
Discussions for another time:
Did I miss anything? Tests:
|
Hey @Eric-Nitschke @ekatef . I think the PR is ready for merge once the above checklist provided by @Eric-Nitschke is done. I also replied to @Eric-Nitschke's comment regarding the config file. Thank you @Eric-Nitschke for the great work and looking forward for future contributions. |
|
Hello 😄 @Eric-Nitschke thanks a lot for all the responses and creating a check-list, thank you so much @Eddy-JV for the in-depth review and providing insights into the architecture 🙏🏽 I wonder what is the plan to proceed? @Eric-Nitschke I see that you have ticked the boxes but it appears that the changes have not been pushed to github. Do you experience any issues with git? [I have no ideas why but git is normally provides you with a lot of surprises when you start working in a repo] Happy to assist in resolving them if needed. Also, I see that there are some conflicts with the main. They result from come changes we introduced recently inside main and can be resolved relatively easily. Let me know if any hints on that would be helpful @Eddy-JV on the architectural level, I understand the idea is to polish the pipelines representation to a level when the model is truly realistic. Though, I have a feeling that the work needed may be considerable. @Eric-Nitschke do you see how to proceed with the advanced implementation proposed by @Eddy-JV? May be any break-down helpful? |
Summary:
Changes:
Reason:
Checklist
envs/environment.yamlanddoc/requirements.txt.config.default.yamlandconfig.tutorial.yaml.test/(note tests are changing the config.tutorial.yaml)doc/configtables/*.csvand line references are adjusted indoc/configuration.rstanddoc/tutorial.rst.doc/release_notes.rstis amended in the format of previous release notes, including reference to the requested PR.