Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

save aligned data for faster data loading #257

Open
krsnapaudel opened this issue Jun 17, 2024 · 1 comment · Fixed by #329
Open

save aligned data for faster data loading #257

krsnapaudel opened this issue Jun 17, 2024 · 1 comment · Fixed by #329

Comments

@krsnapaudel
Copy link
Collaborator

krsnapaudel commented Jun 17, 2024

  1. Align with crop calendar
  2. Align indices of labels and predictors
    I realized that aligning with lead time must be done afterwards. So the current code in trim_to_lead_time must be split into two parts: do 1 and 2 above and save data. Do the trimming to lead time later.
krsnapaudel pushed a commit that referenced this issue Aug 30, 2024
@krsnapaudel krsnapaudel linked a pull request Aug 30, 2024 that will close this issue
@krsnapaudel
Copy link
Collaborator Author

krsnapaudel commented Nov 25, 2024

The code to save and load aligned data was removed in a subsequent merge. Reopening.

Considerations:

  1. Data path vs aligned data. Is the aligned data path inside data path?
  2. Should we reconsider how data is organized now, i.e. crop and then country?
  3. We should also save data after TorchDataset interpolates and aggregates time series. Should we make it a separate issue?

I have second thoughts about saving aligned data because it would be hard to separate aligning to crop season and trimming time series based on lead time.

@krsnapaudel krsnapaudel reopened this Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant