-
Notifications
You must be signed in to change notification settings - Fork 416
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AsyncChunkReader::get_bytes error: Generic MicrosoftAzure error: error decoding response body #2592
Comments
I'm reasonably confident the error is orignating from here, based on my read of various error messages: delta-rs/crates/core/src/operations/optimize.rs Lines 500 to 511 in f041692
Since it's run in a blocking context in the python side, I'm wondering if that's causing any weirdness (it shouldn't). |
@abhiaagarwal I wish I could assist but my Rust knowledge is very limited. But let me know if I need to test something. |
This same issue is happening occasionally when also reading from a deltatable in Azure Gen 2:
In which |
@Josh-Hiz what happens if try benchmarking with |
Very gentle ping @Josh-Hiz :-) |
@thomasfrederikhoeck try to create a reproducible example that mimics the size and characteristics of your table on Azure. Otherwise no one can properly replicate |
@thomasfrederikhoeck can you test in your environment if I provide you a custom wheel? |
@ion-elgreco if you have a branch I can build that and try tmw? |
@thomasfrederikhoeck https://github.com/ion-elgreco/delta-rs/tree/chore/fs_debug Ah wait this is just for reading through pyarrow : P |
@ion-elgreco Hmm I'm not seeing that debug print you added if I run: import os
os.environ["RUST_LOG"]="debug"
os.environ["RUST_BACKTRACE"]="1"
from deltalake import DeltaTable
blob_path = "az://<redacted>"
storage_options = {"AZURE_STORAGE_ACCOUNT_NAME": "<redacted>",
"AZURE_CONTAINER_NAME":'l<redacted>',
'azure_use_azure_cli': 'true',
}
dt = DeltaTable(blob_path, storage_options=storage_options)
dt.optimize.z_order(["<redacted>"]) Do I need to build with certain maturin args? |
Yeah it's only added in reading through pyarrow dataset interface, the issues you see requires some refactoring in delta-rs 😞 |
@thomasfrederikhoeck can you try this one please, it uses an additional runtime for the writing part of optimize: https://github.com/ion-elgreco/delta-rs/tree/fix/use_different_write_rt |
@thomasfrederikhoeck I introduced a separate runtime for IO, can you try 0.19.1 please and let me know if things are improved |
It worked!! Nice job @ion-elgreco! |
@thomasfrederikhoeck Yey :D |
Environment
Delta-rs version: 0.18.1
Binding: Python
Environment:
Bug
What happened:
After 0.18.1 was released it fixes the inital issue with #2301 for me but instead I started hitting this. The Z-order operations start and I can see that there is usage of network, CPU and memory but after 30 secs-ish I'm hit with the following. The Rust logs doens't show anything strange:
What you expected to happen:
That the Z-order completes.
How to reproduce it:
More details:
The text was updated successfully, but these errors were encountered: