Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert strings columns output from cudf::read_json to other types #2510

Draft
wants to merge 37 commits into
base: branch-24.12
Choose a base branch
from
Draft

Convert strings columns output from cudf::read_json to other types #2510

wants to merge 37 commits into from

Conversation

ttnghia
Copy link
Collaborator

@ttnghia ttnghia commented Oct 17, 2024

This implements the conversion from strings columns to other data types to match with the output of from_json in Spark.

Closes #2468.

Contribute to NVIDIA/spark-rapids#11560.

@ttnghia ttnghia requested a review from revans2 October 17, 2024 03:07
@ttnghia ttnghia self-assigned this Oct 17, 2024
@ttnghia ttnghia mentioned this pull request Oct 17, 2024
Open
@ttnghia
Fix empty input
8d7ad2e 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Misc
9e759c4 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add nullifyIfNotQuoted option for removeQuotes
2fff949 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Implement castStringsToDecimals
d09de41 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Copy link
Collaborator Author

ttnghia commented Oct 21, 2024

CC @karthikeyann.

@karthikeyann karthikeyann mentioned this pull request Oct 21, 2024
Open
@ttnghia
WIP
3024583 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
WIP: Implementing fromJSONToStructs
d33d8e2 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Merge branch 'branch-24.12' into convert_table
295c36c
@ttnghia
Fix compile errors
1ea9cc8 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup
c1bb2d4 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Revert code as we still need them
f6634b4
@ttnghia
Add error check
06b2c19 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Add more comments
2dcdd11 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Cleanup
f3c391b 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Return as-is if the column is date/time
52c42a6 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Update test
19c64be 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Merge branch 'branch-24.12' into convert_table
cb9d252
@ttnghia
Update cudf
5d07db1 
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia
Revert "Update cudf"
39e3a9b 
This reverts commit 5d07db1.
@ttnghia
Merge branch 'branch-24.12' into convert_table
8628136
@ttnghia
Update cudf
df1428d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hyperbolic2346 hyperbolic2346 hyperbolic2346 left review comments

@revans2 revans2 Awaiting requested review from revans2

At least 1 approving review is required to merge this pull request.

Assignees

@ttnghia ttnghia

Labels
enhancement New feature or request feature request performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Implement a function to convert a list of strings columns into a structs column with desired schema
2 participants
@ttnghia @hyperbolic2346