Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

don't expand scenario_geography and equity_market until necessary #11

Open
cjyetman opened this issue Feb 27, 2023 · 0 comments
Open

don't expand scenario_geography and equity_market until necessary #11

cjyetman opened this issue Feb 27, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@cjyetman
Copy link
Member

cjyetman commented Feb 27, 2023

https://github.com/RMI-PACTA/pacta.data.preparation/blob/ba0f8b8518afb2d00bfe5d9bff1a935418eaa5dd/R/dataprep_abcd_scen_connection.R#L143-L151

Up until merging in the scenario data, the expansion of the data with the scenario_geography and equity_market columns drastically multiplies the number of rows in the data, and the grouped calculations necessitated by these otherwise duplicated rows is a source of the incredibly long run times. Basically for every combination of id, technology, and year we are multiplying the rows by every combination of scenario_geography and equity_market and calculating duplicate data for all of them.

We should carefully consider if this is actually necessary, and if not calculate as much and we can before expanding to the scenario_geography and equity_market values. @jdhoffa @jacobvjk @AlexAxthelm

related #7

@cjyetman cjyetman added the enhancement New feature or request label Feb 27, 2023
@jdhoffa jdhoffa transferred this issue from another repository Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant