Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support parquet files with nanoseconds timestamps when converting a Parquet table to a Delta table #1721

Open
junjunjd opened this issue Oct 13, 2023 · 3 comments
Labels
enhancement New feature or request help wanted Extra attention is needed
Milestone

Comments

@junjunjd
Copy link
Contributor

junjunjd commented Oct 13, 2023

Description

When converting from ArrowSchema to deltalake::Schema, the try_from method will return an error if the arrow schema has a Timestamp(nanosecond). This causes the convert_to_delta function to fail. It makes sense to support parquet files with nanoseconds timestamps when converting a Parquet table to a Delta table.

Use Case

Convert parquet files in rust/tests/data/delta-1.2.1-only-struct-stats to a Delta table.

Related Issue(s)
#1041, #1682, #1686

@junjunjd junjunjd added the enhancement New feature or request label Oct 13, 2023
@rtyler rtyler added the bug Something isn't working label Oct 26, 2023
@rtyler rtyler added this to the Correct timestamp handling milestone Oct 26, 2023
@hntd187
Copy link
Collaborator

hntd187 commented Oct 27, 2023

Take

@MrPowers
Copy link
Contributor

Sweet, assigned it to you @hntd187

@ion-elgreco ion-elgreco added help wanted Extra attention is needed and removed bug Something isn't working labels Dec 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

5 participants