-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dynamic chunking interface for StoreToZarr #595
Merged
Merged
Changes from 18 commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
14e4fa2
add dynamic_chunking_fn field to storetozarr
cisaacstern 1eda370
test dynamic chunking kws
cisaacstern 1765f67
test dynamic chunking with target chunks raises
cisaacstern 1f16d83
Merge remote-tracking branch 'origin/main' into dynamic-chunks-interface
cisaacstern 74b8ebf
xr.Datasets are more familiar than XarraySchemas, so let users genera…
cisaacstern 952ea57
Merge branch 'main' into dynamic-chunks-interface
cisaacstern 147cd20
fix transforms test for attrs
cisaacstern 9e7dd67
Added integration test for dynamic chunking plugin
jbusecke 9770cf5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] f7aee4e
Expanded docstring
jbusecke 4918ffd
Merge branch 'dynamic-chunks-interface' of https://github.com/pangeo-…
jbusecke 81350ac
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 6d3e965
Test patching runner to not depend on recipes
jbusecke 6403002
Merge branch 'dynamic-chunks-interface' of https://github.com/pangeo-…
jbusecke 0c0819e
Merge remote-tracking branch 'upstream/main' into dynamic-chunks-inte…
jbusecke ad2e237
Did we need that patch at all?
jbusecke 7983643
reverse install order
jbusecke c7c5206
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] d177dc4
Add chunking test
jbusecke cbbfe40
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] dcb1aa7
reduce chunking
jbusecke 9e15a25
Merge branch 'dynamic-chunks-interface' of https://github.com/pangeo-…
jbusecke 7dc9be7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] ff5fa4c
Added checks to non-chunked gpcp recipe and reduced dynamic chunks to 2
jbusecke caef6c0
Merge branch 'dynamic-chunks-interface' of https://github.com/pangeo-…
jbusecke 5ee3f8a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] f19acfb
Add release notes
jbusecke 2afd727
Merge branch 'dynamic-chunks-interface' of https://github.com/pangeo-…
jbusecke File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
from typing import Dict | ||
|
||
import apache_beam as beam | ||
import pandas as pd | ||
import xarray as xr | ||
import zarr | ||
|
||
from pangeo_forge_recipes.patterns import ConcatDim, FilePattern | ||
from pangeo_forge_recipes.transforms import OpenURLWithFSSpec, OpenWithXarray, StoreToZarr | ||
|
||
dates = [ | ||
d.to_pydatetime().strftime("%Y%m%d") | ||
for d in pd.date_range("1996-10-01", "1999-02-01", freq="D") | ||
] | ||
|
||
|
||
def make_url(time): | ||
url_base = "https://storage.googleapis.com/pforge-test-data" | ||
return f"{url_base}/gpcp/v01r03_daily_d{time}.nc" | ||
|
||
|
||
concat_dim = ConcatDim("time", dates, nitems_per_file=1) | ||
pattern = FilePattern(make_url, concat_dim) | ||
|
||
|
||
def test_ds(store: zarr.storage.FSStore) -> zarr.storage.FSStore: | ||
# This fails integration test if not imported here | ||
# TODO: see if --setup-file option for runner fixes this | ||
import xarray as xr | ||
|
||
ds = xr.open_dataset(store, engine="zarr", chunks={}) | ||
assert ds.title == ( | ||
"Global Precipitation Climatatology Project (GPCP) " "Climate Data Record (CDR), Daily V1.3" | ||
) | ||
return store | ||
|
||
|
||
def chunk_func(ds: xr.Dataset) -> Dict[str, int]: | ||
return {"time": 3} | ||
|
||
|
||
recipe = ( | ||
beam.Create(pattern.items()) | ||
| OpenURLWithFSSpec() | ||
| OpenWithXarray(file_type=pattern.file_type, xarray_open_kwargs={"decode_coords": "all"}) | ||
| StoreToZarr( | ||
dynamic_chunking_fn=chunk_func, | ||
store_name="gpcp.zarr", | ||
combine_dims=pattern.combine_dim_keys, | ||
) | ||
| "Test dataset" >> beam.Map(test_ds) | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jbusecke let's assert that the chunking of the opened dataset is in fact what we set dynamically in with the dynamic chunking function :)