feat: Support S3 paths in Parquet inputs #798

bjchambers · 2023-10-07T04:06:45Z

Remaining work for S3 support includes preparing to S3 and writing to S3 destinations.

.github/workflows/ci_python.yml

crates/sparrow-main/tests/e2e/parquet_tests.rs

crates/sparrow-runtime/src/stores/object_store_url.rs

python/pysrc/kaskada/sources/arrow.py

python/src/session.rs

python/src/source.rs

python/src/table.rs

feat: Support S3 paths in Parquet inputs

e6e8e96

Remaining work for S3 support includes preparing to S3 and writing to S3 destinations.

bjchambers requested a review from jordanrfrazier October 7, 2023 04:06

cla-bot bot added the cla-signed Set when all authors of a PR have signed our CLA label Oct 7, 2023

github-actions bot added build enhancement New feature or request sparrow labels Oct 7, 2023

bjchambers requested a review from epinzur October 7, 2023 04:09

epinzur approved these changes Oct 7, 2023

View reviewed changes

jordanrfrazier reviewed Oct 7, 2023

View reviewed changes

comments

e40bdca

jordanrfrazier approved these changes Oct 9, 2023

View reviewed changes

python/src/source.rs Outdated Show resolved Hide resolved

python/src/table.rs Outdated Show resolved Hide resolved

bjchambers added 2 commits October 9, 2023 10:44

lint

ba162a2

rename url to file

4a9758a

bjchambers enabled auto-merge October 9, 2023 17:52

fix deps

1231e23

bjchambers added this pull request to the merge queue Oct 9, 2023

Merged via the queue into main with commit d893f39 Oct 9, 2023
31 checks passed

bjchambers deleted the ben/parquet-read-s3 branch October 9, 2023 22:37

Provide feedback