Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An ability to add goodtables checks to the validate processor #142

Open
roll opened this issue Jun 12, 2020 · 2 comments
Open

An ability to add goodtables checks to the validate processor #142

roll opened this issue Jun 12, 2020 · 2 comments

Comments

@roll
Copy link
Contributor

roll commented Jun 12, 2020

Overview

From goodtabes@3 (now early alpha) it will have a function like system.create_check('baseline/integrity/etc') returning a check object with check.validate_headers/row/table available. The amount of checks have been drastically reduces (only 2 core checks and 6 advanced).

So we will be able to integrate it into the validate processor like (maybe adding core checks baseline/integrity by default):

Flow(
    validate(extra_checks=['baseline', 'integrity', ('blacklisted-value', {'fieldName': 'name'}), ...]
)

As an output, I think we just can add the errors list to a resource descriptor with a list of found errors.

Initially dataflows-goodtables integration is a @cschloer's idea but, for now, I don't see how we can friend them due to their streaming-nature without going to lower-level (the level of the individual check).

@cschloer
Copy link
Contributor

Hey, is our best bet on this just to wait until frictionless-py is fully implemented and dataflows has migrated?

@roll
Copy link
Contributor Author

roll commented Nov 1, 2020

Yea. I think so. Now this atomic checks architecture works with Frictionless so we can apply checks on streams (as a dataflows processor without consuming a stream separately)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants