Description
Is your feature request related to a problem?
Xarray maintains the encoding of datasets read from most of its supported backend formats (e.g. NetCDF, Zarr, etc.). This is very useful when you want to perfectly roundtrip but it often gets in the way, causing conflicts when writing a modified dataset or when appending to another dataset. Most of the time, the solution is to just remove the encoding from the dataset and continue on. The following code sample is found in a number of issues that reference this problem.
for v in list(ds.coords.keys()):
if ds.coords[v].dtype == object:
ds[v].encoding.clear()
for v in list(ds.variables.keys()):
if ds[v].dtype == object:
ds[v].encoding.clear()A sample of issues that show variants of this problem.
- Error when writing string coordinate variables to zarr #3476
- ValueError when trying to encode time variable in a NetCDF file with CF convensions #3739
- Error when rechunking from Zarr store #4380
- Zarr encoding attributes persist after slicing data, raising error on
to_zarr#5219 to_zarr(append_dim="time")appends incorrect datetimes #5969to_zarrwith append or region mode and_FillValuedoesnt work #6329- to_netcdf from subsetted Dataset with strings loaded from char array netCDF can sometimes fail #6352
Describe the solution you'd like
In many cases, the solution to these problems is to leave the original dataset encoding behind and either use Xarray's default encoding (or the backends default) or to specify one's own encoding options. Both cases would benefit from a convenience method to reset the original encoding. Something like would serve this process:
ds = xr.open_dataset(...).reset_encoding()Describe alternatives you've considered
Variations on the API above could also be considered:
xr.open_dataset(..., keep_encoding=False)or even:
with xr.set_options(keep_encoding=False):
ds = xr.open_dataset(...)We can/should also do a better job of surfacing inconsistent encoding in our backends (e.g. to_netcdf).
Additional context
No response