Datapipe State Transition Hardening and ADCS Post Processing Fixes

[zinic]

Description

This adds an ingest task lookup before transitioning file upload jobs from the INGESTING state to the ANALYZING state. There is a potential race where a job may transition into INGESTING after ingestAvailableTasks(...) is called. In this rare case, ingest tasks would remain until the next datapipe loop tick. Upon ingestion of the remaining tasks analysis would not be triggered, leaving the system in a inconsistent state until the next analysis run.

Additionally, this changeset contains a change in the ADCS post-processing logic that defers creation of a parallel operation context. The functions that previously ran between creation of the parallel operation context would spin up their own parallel operation contexts resulting in creation of 12+ concurrent connections to the database. The system is designed to limit concurrent connections to 10 leaving some of the parallel operations hung until their association context is canceled.

Motivation and Context

State transition hardening will ensure there are no graph inconsistency gaps. The ADCS post-processing change will increase resiliency of the associated parallel operation code paths.

How Has This Been Tested?

Minor changes are covered by existing integration tests.

Types of changes

• Chore (a change that does not modify the application functionality)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Documentation updates are needed, and have been made accordingly.
• I have added and/or updated tests to cover my changes.
• All new and existing tests passed.
• My changes include a database migration.

[computator]

I don't know the details of the ADCS logic but the rest looks good!

[rvazarkar]

lgtm

[urangel]

Thanks for the adcs post cleanup!