Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bueno change: Handle multiple file inputs for Plugins #63

Open
qwofford opened this issue Jul 25, 2023 · 0 comments
Open

Bueno change: Handle multiple file inputs for Plugins #63

qwofford opened this issue Jul 25, 2023 · 0 comments
Assignees

Comments

@qwofford
Copy link
Collaborator

Currently the Bueno plugin handles the case where a single file is responsible for determining the schema and sampling metadata.

We need to support a list of files. The Plugin should build up the schema from the union of input files, and append one or more rows for each file parsed. If any row values are missing, we should fill them with null values.

At the end of a transload operation, we should report the following to the user:

  • Nothing if null values were not inserted
  • If null values are inserted, report a warning: the number of rows, the number of columns, the number of null values inserted, the percentage of rows and values which are null. A high percentage indicates that the files have very different contents, and may require different plugins or special handling logic in the existing plugin.
@qwofford qwofford self-assigned this Jul 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant