Skip to content

Rule and DataFrame datatypes not matching #105

Closed Answered by canimus
FShinoda asked this question in Q&A
Discussion options

You must be logged in to vote

Hello, happy to help. What kind of check are you performing? I assume that is has_pattern as it is the only one with a string requirement.
And yes, certain checks are associated to specific data types as shown in the README. One alternative is to simply cast your column:

# assuming your dataframe as `acc_benefits_id`
from cuallee import Check, CheckLevel
check = Check(CheckLevel.WARNING, "Pattern")
check.has_pattern("acc_benefits_id_str", "your_pattern")
check.validate(df.select("acc_benefits_id").cast("string").alias("acc_benefits_id_str")).show()

However, I do think that a pattern check in a numeric data type, is probably not what you are searching for isn't?
Is computationally more exp…

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by FShinoda
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants