chore: simplify datapipe and test orphan file cleanup

[zinic]

Description

This is a continuation of work done earlier. Mostly clean up and a small refactor to pull additional complexity from the datapipe and add test coverage.

Motivation and Context

Further clean up from earlier datapipe refactors.

How Has This Been Tested?

Added additional unit tests to cover this refactor.

Types of changes

• Chore (a change that does not modify the application functionality)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Documentation updates are needed, and have been made accordingly.
• I have added and/or updated tests to cover my changes.
• All new and existing tests passed.
• My changes include a database migration.

[rvazarkar]

Looks good, and extra testing is always great. A comment explaining how the ExpectedFile test works would be great as its not immediately obvious (at least to me)