Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

seasonal PSM calibration takes a long time #20

Open
frodre opened this issue Feb 15, 2018 · 1 comment
Open

seasonal PSM calibration takes a long time #20

frodre opened this issue Feb 15, 2018 · 1 comment
Assignees

Comments

@frodre
Copy link
Collaborator

frodre commented Feb 15, 2018

I believe seasonal PSM calibration takes a long time because it is reloading the calibration dataset for each season defined. This can be fixed by holding the original calibration in memory, or if it's too big, resorting to using Dask to run the re-averaging operations. In my experience Dask has only worked with HDF5 files, but since we've upgraded to Python 3 this may be fixed for netCDF.

@rtardif Trying to configure just like P2B now that my seasonal PSMs are built (31hrs!)-- do I set all proxy_psm_type to bilinear in the config yml file, like this: -- dmanderson

@frodre frodre self-assigned this Feb 15, 2018
@brews
Copy link
Collaborator

brews commented Feb 16, 2018

xarray chunking might be an easy solution for the netcdf files? (uses dask under the hood) -- just a thought.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants