Feature: Filter out records with invalid relations

[csun-cpointe]

Description

If a reference record has an invalid field, we would like to filter out the base record from the dataset. All other base records with valid reference data should be preserved.

DOD

Acceptance criteria required to realize the requested feature

• Validate reference record fields
• Validate that the required reference records are present
• Remove the base record when a reference record is invalid or missing when required
• Preserve the valid base record

BDD Scenarios:

  Scenario: An 1-1 or M-1 relation data record that has invalid data is removed

  Scenario: An 1-M relation data record that has invalid data is removed

  Scenario: A required 1-1 or M-1 relation data record that is not set is removed

  Scenario: A non-required 1-1 or M-1 relation data record that is not set is preserved

  Scenario: A required 1-M relation data record that is empty is removed

  Scenario: A required 1-M relation data record that is not set is removed

  Scenario: A non-required 1-M relation data record that is not set is preserved

Test Strategy/Script

How will this feature be verified?
The above BDD scenarios should fully test the added capability, so a passing build should suffice for verification.

References/Additional Context

A clear and concise description of any alternative solutions or features you've considered.
Add any other context, links, or screenshots about the feature request here.

• After the validation, the dataset schema should remain the same
• We should only use the spark transformation over the action when validating the dataset
• Look into better performance implementation. i.e.: UDF function vs custom transformation
• We will not implement the notification for the invalid records because without actual data processing, we will not able to detect the invalid record. This contradicts the goal of using only spark transformation

[csun-cpointe]

DoD completed with @ewilkins-csi and @nartieri

[csun-cpointe]

OTS completed with @carter-cundiff

[carter-cundiff]

Testing passed: