How may I register a parquet file on an s3 bucket? #7109
-
|
How may I register a parquet file using an s3 url? Locally I could do ctx.register_parquet("demo", "/Users/me/Downloads/dlake/data/table/1-46631076-a41d-4305-8de2-71f5e65158aa-0.parquet", ParquetReadOptions::default()).await?; On the other hand, this does not work: ctx.register_parquet("demo", "s3://data/table1/1-46631076-a41d-4305-8de2-71f5e65158aa-0.parquet", ParquetReadOptions::default()).await?; Fails with error Error: Internal error: No suitable object store found for s3://data/table1/1-46631076-a41d-4305-8de2-71f5e65158aa-0.parquet. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
You need to register the s3 bucket first. See https://github.com/apache/arrow-datafusion/blob/main/datafusion-examples/examples/query-aws-s3.rs for a fully worked example of reading from S3. |
Beta Was this translation helpful? Give feedback.
You need to register the s3 bucket first. See https://github.com/apache/arrow-datafusion/blob/main/datafusion-examples/examples/query-aws-s3.rs for a fully worked example of reading from S3.