Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dask for coregistration #525

Draft
wants to merge 23 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
c193b20
feat: preprocess_coreg for xarray input
ameliefroessl Apr 30, 2024
e54fa12
feat: subsample with dask array
ameliefroessl May 21, 2024
70d48e7
tests: test biascorr synthetic data for dask/ xarray
ameliefroessl May 22, 2024
85b54ef
fix: calculating mask on the correct input
ameliefroessl May 31, 2024
f5a75e5
refactor: clearer mask
ameliefroessl May 31, 2024
43bb825
feat: dask version of meshgrid
ameliefroessl May 31, 2024
f5778ed
feat: map_blocks for polyval2d
ameliefroessl May 31, 2024
74e0981
feat: apply() adapted to dask input
ameliefroessl Jun 4, 2024
6be8a15
fix: precommit
ameliefroessl Jun 4, 2024
044d6f0
refactor: rename postprocessing function
ameliefroessl Jun 5, 2024
d88f09e
fix: remove explicit cog paths and wrong default parameter
ameliefroessl Jun 5, 2024
9f5518c
tests: create raster mask on the fly from vector data in xarray test …
ameliefroessl Jun 6, 2024
110c6cb
fix: correct type for fit function parameters
ameliefroessl Jun 6, 2024
b46d20c
fix: error with weights
ameliefroessl Jun 6, 2024
0452bd8
fix: calculate valid_mask from bias_var
ameliefroessl Jun 7, 2024
1d74be6
refactor: generalized wrapper for fit functions to evaluate them chunked
ameliefroessl Jun 10, 2024
7ce44d1
feat: map logic for _postprocess_coreg_apply_xarray
ameliefroessl Jun 10, 2024
cb07a80
tests: remove unused mask_cog file reference
ameliefroessl Jun 10, 2024
1c94c72
docs: cleaning up a bit
ameliefroessl Jun 11, 2024
67a432a
docs: correct type hints
ameliefroessl Jun 11, 2024
624d69b
refactor: get_valid_data, mask_data and valid_data_darr for generaliz…
ameliefroessl Jun 11, 2024
3adb1b6
fix: fixing pytest
ameliefroessl Jun 14, 2024
1c57a38
refactor: moving dask check out of if for general biascorr dask
ameliefroessl Jun 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: map logic for _postprocess_coreg_apply_xarray
ameliefroessl committed Jun 10, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 7ce44d1c61c8ed7f8057f0fca8e4fc5729443e25
52 changes: 48 additions & 4 deletions xdem/coreg/base.py
Original file line number Diff line number Diff line change
@@ -724,13 +724,51 @@ def _postprocess_coreg_apply_pts(


def _postprocess_coreg_apply_xarray(
applied_elev: da.Array, out_transform: affine.Affine
elev: da.Array,
applied_elev: da.Array,
transform: affine.Affine,
out_transform: affine.Affine,
crs: rio.crs.CRS,
resample: bool,
resampling: rio.warp.Resampling | None = None,
) -> tuple[da.Array, affine.Affine]:
"""Post-processing and checks of apply for dask inputs."""

# TODO mimic what is happening in the postprocess_coreg_apply_rst.
# make sure the datatype is correct
if applied_elev.dtype != np.float32:
applied_elev = applied_elev.astype(np.float32)

# Reproject the corrected elevation
# NOTE is there a way to make this optional? It can save some compute time.
reprojected = delayed_reproject(
darr=applied_elev,
src_transform=out_transform,
src_crs=crs,
dst_transform=transform,
dst_shape=elev.shape,
dst_crs=crs,
resampling=resampling,
src_nodata=np.nan,
dst_nodata=elev.rio.nodata,
dst_chunksizes=None,
)

return applied_elev, out_transform
# Set nans to nodata value
reprojected = da.where(da.isnan(reprojected), elev.rio.nodata, reprojected)

output_ds = DataArray(
da.expand_dims(reprojected, axis=0),
coords=elev.coords, # TODO is this correct?
dims=["band", "y", "x"],
name="Corrected DEM",
# attrs={}, # it's possible to set geotiff metadata via the attrs parameter
)

# Set crs and nodata value
output_ds = output_ds.rio.set_crs(elev.rio.crs)
output_ds = output_ds.rio.set_nodata(elev.rio.nodata)

return output_ds, out_transform


def _postprocess_coreg_apply_rst(
@@ -820,7 +858,13 @@ def _postprocess_coreg_apply(
)
elif isinstance(applied_elev, da.Array):
applied_elev, out_transform = _postprocess_coreg_apply_xarray(
applied_elev=applied_elev, out_transform=out_transform
elev=elev,
applied_elev=applied_elev,
transform=transform,
crs=crs,
out_transform=out_transform,
resample=resample,
resampling=resampling,
)

else: