Skip to content

datafusion-cli: Use correct S3 region if it is not specified #16502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

liamzwbao
Copy link
Contributor

@liamzwbao liamzwbao commented Jun 22, 2025

Which issue does this PR close?

Rationale for this change

Improve UX by resolving s3 region automatically.

What changes are included in this PR?

  • Resolve bucket region if not specified
  • Retry on wrong region setting

Are these changes tested?

  • unit tests for resolving region
  • integration tests against a public bucket

Are there any user-facing changes?

Yes, the CLI can automatically resolve the region for bucket now

@liamzwbao liamzwbao force-pushed the issue-13456-resolve-region-on-error branch 4 times, most recently from fa796a8 to 1800fc4 Compare June 24, 2025 23:34
@liamzwbao liamzwbao force-pushed the issue-13456-resolve-region-on-error branch from 1800fc4 to 126b9e1 Compare June 25, 2025 00:31
@liamzwbao liamzwbao marked this pull request as ready for review June 25, 2025 00:44
@liamzwbao
Copy link
Contributor Author

Hi @alamb, this PR is ready for review!

One concern here is whether it is acceptable to test against a bucket outside of datafusion, but I couldn’t find a good way to test region retry against local minio.

@alamb
Copy link
Contributor

alamb commented Jun 25, 2025

Thanks @liamzwbao -- I'll put this one on my review queue for tomorrow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

datafusion-cli: Use correct S3 region if it is not specified
2 participants