Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling vintage availability #10

Open
brynpickering opened this issue Feb 23, 2024 · 6 comments
Open

Handling vintage availability #10

brynpickering opened this issue Feb 23, 2024 · 6 comments
Labels
enhancement New feature or request

Comments

@brynpickering
Copy link
Member

What can be improved?

We have to currently input vintage availability manually. Ideally we would input it programatically as part of the math.

I have put together two possible ways we can do it, with different math in the YAML vs in the helper functions. @irm-codebase which do you think is more readable?

Both have this new math to get the year as an integer from the vintagesteps/investsteps (necessary as long as this is still an issue: calliope-project/calliope#567).

YAML math

global_expressions:
  investment_year:
    foreach: [investsteps]
    equations:
      - expression: year(investsteps)

  vintage_year:
    foreach: [vintagesteps]
    equations:
      - expression: year(vintagesteps)
      
  investstep_resolution:
    foreach: [investsteps]
    equations:
      - where: investsteps=get_val_at_index(investsteps, 0)
        expression: get_val_at_index(investment_year, 1) - initial_year
      - where: NOT investsteps=get_val_at_index(investsteps, 0)
        expression: investment_year - roll(investment_year, investsteps=1)

Helper functions

class Year(ParsingHelperFunction):
    #:
    NAME = "year"
    #:
    ALLOWED_IN = ["where", "expression"]

    def as_math_string(self, array: str) -> str:
        return f"year({array})"

    def as_array(self, array: xr.DataArray) -> xr.DataArray:
        return array.dt.year

version 1:

YAML math

global_expressions:
  available_vintages:
    foreach: [vintagesteps, investsteps, techs]
    equations:
      - expression: get_available_vintages(weibull)

Helper functions

class GetVintageAvailability(ParsingHelperFunction):
    #:
    NAME = "get_vintage_availability"
    #:
    ALLOWED_IN = ["expression"]

    def _weibull_func(self, year_diff: xr.DataArray) -> xr.DataArray:
        shape = self._input_data.get("shape", 1)
        gamma = scipy.special.gamma(1 + 1 / shape)
        availability = np.exp(
            -((year_diff / self._input_data.lifetime.fillna(np.inf)) ** shape)
            * (gamma**shape)
        )
        return availability.fillna(0)

    def _linear_func(self, year_diff: xr.DataArray) -> xr.DataArray:
        availability = 1 - (year_diff / self._input_data.lifetime)
        return availability.clip(min=0)

    def _step_func(self, year_diff: xr.DataArray) -> xr.DataArray:
        life_diff = self._input_data.lifetime - year_diff
        availability = (life_diff).clip(min=0) / life_diff
        return availability
    
    def as_math_string(self, method: Literal["weibull", "linear", "step"]) -> str:
        # TODO: implement for each func type
        pass

    def as_array(self, method: Literal["weibull", "linear", "step"]) -> xr.DataArray:
        """For each investment step in pathway optimisation, get the historical capacity additions that now must be decommissioned.

        Args:
            method (str): The method with which to assume technology survival rates.

        Returns:
            xr.DataArray:
        """
        year_diff = (
            self._input_data.investsteps.dt.year - self._input_data.vintagesteps.dt.year
        )
        year_diff_no_negative = year_diff.where(year_diff >= 0)
        if method == "weibull":
            availability = self._weibull_func(year_diff_no_negative)
        elif method == "linear":
            availability = self._linear_func(year_diff_no_negative)
        elif method == "step":
            availability = self._step_func(year_diff_no_negative)
        else:
            raise ValueError(f"Cannot get vintage availability with `method`: {method}")
        return availability.where(year_diff_no_negative.notnull())

version 2:

YAML math

global_expressions:
  available_vintages:
    foreach: [vintagesteps, investsteps, techs]
    equations:
      - where: config.vintage_survival=weibull
        expression: >-
          exponential(
            -(($year_diff / default_if_empty(lifetime, inf)) ** shape)
            * (gamma(1 + 1 / shape) ** shape)
          )
      - where: config.vintage_survival=linear
        expression: >-
           clip(1 - ($year_diff / lifetime), lower=0)
      - where: config.vintage_survival=step
        expression: >-
          clip(lifetime - $year_diff, lower=0) / (lifetime - $year_diff)
    sub_expressions:
      year_diff:
        - expression: investment_year - vintage_year

Helper functions

class Exponential(ParsingHelperFunction):
    #:
    NAME = "exponential"
    #:
    ALLOWED_IN = ["expression"]

    def as_math_string(self, array: str) -> str:
        return rf"\exp^{{{array}}}"

    def as_array(self, array: xr.DataArray) -> xr.DataArray:
        return np.exp(array)


class Gamma(ParsingHelperFunction):
    #:
    NAME = "gamma"
    #:
    ALLOWED_IN = ["expression"]

    def as_math_string(self, array: str) -> str:
        return rf"\Gamma({array})"

    def as_array(self, array: xr.DataArray) -> xr.DataArray:
        return scipy.special.gamma(array)

class Clip(ParsingHelperFunction):
    #:
    NAME = "clip"
    #:
    ALLOWED_IN = ["expression"]

    def as_math_string(self, array: str, lower: Optional[str] = None, upper: Optional[str] = None) -> str:
        base = rf"\text{{clip}}({array}"
        if lower is not None:
            base += rf", \text{{lower}}={lower}"
        if upper is not None:
            base += rf", \text{{upper}}={upper}"
        return base + ")"

    def as_array(self, array: xr.DataArray, lower: Optional[str] = None, upper: Optional[str] = None) -> xr.DataArray:
        return array.clip(min=lower, max=upper)

Version

v0.1.0

@brynpickering brynpickering added the enhancement New feature or request label Feb 23, 2024
@irm-codebase
Copy link
Collaborator

This is fantastic!

For now, I think I prefer version 1. My reasoning:

  • Very easy to read on both sides, you only need to "jump" between files once.
  • Once the user has familiarized themselves with the different names of the options in get_available_vintages, it should be very easy to read/debug.
  • Easier to upgrade in the future. Just add an extra function on the python side.

Ideally, the yaml configuration should be able to support any and all kinds of math. But this is a very hard milestone that will need a lot of careful thought. Version 1 will make that "jump" easier than version 2 because all the complexity is on the side that would be replaced.

Basically, version 2 still needs jumping to the python files, so it's just as complex (read-wise) as version 1, but with less flexibility and harder debugging.

@irm-codebase
Copy link
Collaborator

irm-codebase commented Feb 26, 2024

Another comment:

In theory, we do not really "need" the investstep or vintagestep for this. Instead, what we need is the difference between them. If it was possible to poll these parameters using that difference when constructing the math, you would only need to define the trend once, and not for each investstep/vintagestep relation

This can save memory (and make things easier to understand) when you have very fine investment resolutions.
See the example below for our case between 2020 and 2050 when there are only 1 step jumps:

vintagesteps 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2042 2043 2044 2045 2046 2047 2048 2049 2050 2043 2044 2045 2046 2047 2048 2049 2050 2044 2045 2046 2047 2048 2049 2050 2045 2046 2047 2048 2049 2050 2046 2047 2048 2049 2050 2047 2048 2049 2050 2048 2049 2050 2049 2050 2050
investsteps 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2020 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2021 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2022 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2023 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2024 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2025 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2027 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2028 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2029 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2030 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2031 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2032 2033 2033 2033 2033 2033 2033 2033 2033 2033 2033 2033 2033 2033 2033 2033 2033 2033 2033 2034 2034 2034 2034 2034 2034 2034 2034 2034 2034 2034 2034 2034 2034 2034 2034 2034 2035 2035 2035 2035 2035 2035 2035 2035 2035 2035 2035 2035 2035 2035 2035 2035 2036 2036 2036 2036 2036 2036 2036 2036 2036 2036 2036 2036 2036 2036 2036 2037 2037 2037 2037 2037 2037 2037 2037 2037 2037 2037 2037 2037 2037 2038 2038 2038 2038 2038 2038 2038 2038 2038 2038 2038 2038 2038 2039 2039 2039 2039 2039 2039 2039 2039 2039 2039 2039 2039 2040 2040 2040 2040 2040 2040 2040 2040 2040 2040 2040 2041 2041 2041 2041 2041 2041 2041 2041 2041 2041 2042 2042 2042 2042 2042 2042 2042 2042 2042 2043 2043 2043 2043 2043 2043 2043 2043 2044 2044 2044 2044 2044 2044 2044 2045 2045 2045 2045 2045 2045 2046 2046 2046 2046 2046 2047 2047 2047 2047 2048 2048 2048 2049 2049 2050

Into this:

age 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

This would be useful since the available_vintages parameter would have the same size as available_initial_cap, unless for some reason it improves over time.

@brynpickering
Copy link
Member Author

It's never going to be a particularly large array (timesteps will always win there) so I'm not really concerned about memory. I'm also unsure how age would be used in the math. You need some way to link when a technology was deployed to know its age, so in every investstep you'd have to have some way to link quantities of capacity with age. And then you'd also lose the option in future to differentiate between the same technology with the same age but different deployment years (a 10-year-old technology deployed in 2020 may have a different set of characteristics to the same tech deployed in 2040)

@brynpickering
Copy link
Member Author

This is fantastic!

For now, I think I prefer version 1. My reasoning:

* Very easy to read on both sides, you only need to "jump" between files once.

* Once the user has familiarized themselves with the different names of the options in `get_available_vintages`, it should be very easy to read/debug.

* Easier to upgrade in the future. Just add an extra function on the python side.

Ideally, the yaml configuration should be able to support any and all kinds of math. But this is a very hard milestone that will need a lot of careful thought. Version 1 will make that "jump" easier than version 2 because all the complexity is on the side that would be replaced.

Basically, version 2 still needs jumping to the python files, so it's just as complex (read-wise) as version 1, but with less flexibility and harder debugging.

Makes sense, although with every function the user has to define the LaTex math representation, which you mostly get for free with the YAML math. It's also not as easy to find the helper functions. We can add them to the documentation, but there is a reason we've moved things to YAML; it's easier to read and edit.

For now though, we'll go with 1. I wonder whether just allowing the numpy library to be accessed within YAML math expressions (numpy.exp(...), numpy.random.gamma(...)) would be sufficient to avoid custom helper functions entirely...

@irm-codebase
Copy link
Collaborator

irm-codebase commented Feb 27, 2024

It's never going to be a particularly large array (timesteps will always win there) so I'm not really concerned about memory. I'm also unsure how age would be used in the math. You need some way to link when a technology was deployed to know its age, so in every investstep you'd have to have some way to link quantities of capacity with age. And then you'd also lose the option in future to differentiate between the same technology with the same age but different deployment years (a 10-year-old technology deployed in 2020 may have a different set of characteristics to the same tech deployed in 2040)

tl;dr: I think the difference in this case depends on how readable we want this to be on the user side.

You are right. This won't use too much memory anyhow, so that's not a good argument on my side.

However, I think it's important to consider readability. If there is something odd going on, checking a single line instead of a sequence on two dimensions is easier. However, this "age" suggestion would only really work if the sum within the constraint has limits on range (sum from investment zero until the current one). Sort of like this (in pyomo):

def c_cap_transfer(model: pyo.ConcreteModel, tech: str, investstep: int):
    """Transfer installed capacity between year slices."""
    avail_new_cap = sum(model.cnew[tech, v]*model.available_vintages[tech, investstep-v] for v in model.Investsteps if v <= investstep)
    avail_ini_cap = model.inicap[tech]*model.available_initial_cap[tech, investstep]
    return model.ctot[tech, investstep] == avail_ini_cap + avail_new_cap

If you want the available_vintages to vary between deployment years, then you'd have to also include the vintages in the parameter, like we are currently doing (model.available_vintages[tech, v, investstep-v]). The rest should remain the same, replicating the current approach to parameter specificity (global tech param -> node-specific tech param; global age param-> vintage age param).

However, if other parameters related to the technology vary too, we will need a new constraint because you can no longer aggregate all years into a single total capacity (unless you give it a vintage index too, and we want to avoid that in the base case). That will happen regardless of how we approach this issue.

I guess the current approach is fine, but it might need revisions if other "age" aspects are introduced...

@irm-codebase
Copy link
Collaborator

irm-codebase commented Feb 27, 2024

Makes sense, although with every function the user has to define the LaTex math representation, which you mostly get for free with the YAML math. It's also not as easy to find the helper functions. We can add them to the documentation, but there is a reason we've moved things to YAML; it's easier to read and edit.

For now though, we'll go with 1. I wonder whether just allowing the numpy library to be accessed within YAML math expressions (numpy.exp(...), numpy.random.gamma(...)) would be sufficient to avoid custom helper functions entirely...

Ufff... you are right.

I agree that helper functions are hard to find... I struggled a bit to find the ones that shift timesteps. So maybe solution 2 is better long-term. Especially if it avoids having to re-define math for LaTeX documentation.

The numpy comment sounds interesting, but would that interact well with LaTeX conversion?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants