feat: adding functionality to run quality checks over streaming sources #19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- changed core functionality of the framework to implement streaming processing - added separate implementations of metric processor: for static and streaming data sources - added streaming DQ job and added corresponding builders in DQ context - refactored error accumulators - added functionality to read streaming virtual sources - other changes connected to implementation of the streaming processing functionality and minor bug fixes.
- documentation updates include description of functionality related to running quality checks over streaming sources - In addition, logging have been revised: Checkita uses Log4J2 library. For earlier versions of Spark (that work with Log4J 1x) the Log4j2 dependency is added explicitly as well as log4j2 -> slf4j bridge.
# Conflicts: # project/Utils.scala
cibaa-team-user
approved these changes
Dec 8, 2023
|
🎉 This PR is included in version 1.1.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
of metric processor: for static and streaming data sources.
Sources that are not streamable are marked accordingly.
Virutal Source readers updated accordingly.
the Log4j2 dependency is added explicitly as well as log4j2 -> slf4j bridge.