-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Identify corrupt data source in odc.stac.load #97
Comments
|
Thanks for the quick response! I'm currently working with an internally hosted and maintained STAC catalog, where sometimes hrefs of assets are broken. That's why it would be handy, if in case a As suggested, I tried to get this information with try:
data = load(items, bands=["green"], chunks={"x": 256, "y": 256})
res = data.sum(dim="time").compute()
except rasterio.errors.RasterioIOError as e:
print(f"e.filename") ... which unfortunately is I guess it would be cool, if except rasterio.errors.RasterioIOError as e:
logger.error(f"Unable to open {asset.href}: \n {e.characters_written}")
raise e |
it is import rasterio
import logging
logging.basicConfig(level="INFO")
logging.getLogger("rasterio").setLevel(logging.DEBUG)
try:
x = rasterio.open("bad_tif.tif")
except rasterio.errors.RasterioIOError as e:
print(f"{e}, {e.filename}")
And also 256px chunks are way too tiny, I recommend starting with 2048 and only going down from that in special situations. |
stac item and asset information are all gone by the time loading is happening inside |
Got it! I will activate more detailed Thanks again for the detailed explanation and keep up the great work! |
@Kirill888 Bringing this up again, (and also in the context of #54). Any chance to implement something similar in odc-stack? |
Thanks! Don't know how I missed that.
One thing that's missing is what does it mean to not fail on error? Will
the band be there? Will it have some nan values? Or something else?
…On Fri, Jul 28, 2023, 08:01 Kirill Kouzoubov ***@***.***> wrote:
@idantene <https://github.com/idantene> similar option is available in
the current release of odc-stac,
fail_on_error=False,
Failed locations are logged with python warning system, see #99
<#99>
—
Reply to this email directly, view it on GitHub
<#97 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AC46YKRBKDKRRGPUD4RQDY3XSNBSFANCNFSM6AAAAAASKGOEI4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
so if your problem is due to broken network connection for example, then you will get back array full of |
There is no way to distinguish between pixels that were missing from the original data and pixels that failed to read, both types end up with
|
Thanks @Kirill888, all of that makes a lot of sense! I wish it was more explicitly mentioned in the documentation, and I still hope to put in a PR for documentation in the future :) |
Hi all, first of all thanks for this great tool!
I'm currently trying to aggregate raster values of a
xarray.DataSet
created withodc-stac
based on several hundred STAC items, similar to this reproducible example:If, however, one of the many items is corrupt, it is very hard to indentify the faulty data source just from the
RasterioIOError
that gets returned:Excecuting the code above returns:
Is there an option to extend the logging in
odc.stac.load
in order to identify which Item/Assetrasterio
wasn't able to open?The text was updated successfully, but these errors were encountered: