BFD-3808: Pipeline submits its own per-dataset metrics to CloudWatch

[malessi]

JIRA Ticket:
BFD-3808

What Does This PR Do?

This PR introduces two new per-dataset, dimensioned CloudWatch Metrics. These new Metrics measure the total time it takes the CCW Pipeline to load an entire RIF dataset as well the active, cumulative time it takes to load. Of particular interest are the following Metrics:

• CcwRifLoadJob.dataset_processing.total.max
• CcwRifLoadJob.dataset_processing.active.duration

The first Metric measures the final, total time to load a given dataset and is submitted once only after the dataset is fully loaded. The second Metric is measured and submitted every minute during the processing of a dataset and actively measures the time the CCW Pipeline is taking to process a dataset. The former will be used in our Dashboards whereas the latter will be used to implement the Time to ingest a weekly batch load once available in S3 SLO Alarms.

What Should Reviewers Watch For?

If you're reviewing this PR, please check for these things in particular:

What Security Implications Does This PR Have?

Please indicate if this PR does any of the following:

• Adds any new software dependencies

• Modifies any security controls

• Adds new transmission or storage of data

• Any other changes that could possibly affect security?

• I have considered the above security implications as it relates to this PR. (If one or more of the above apply, it cannot be merged without the ISSO or team security engineer's (@sb-benohe) approval.)

Validation

Have you fully verified and tested these changes? Is the acceptance criteria met? Please provide reproducible testing instructions, code snippets, or screenshots as applicable.

• Running a CCW Pipeline connected to the 3808-test environment, introducing a synthetic and non-synthetic load in S3, and verifying that:
  • the aforementioned Metrics are submitted to CloudWatch
  • all Metric values are submitted
• Running the 2 new IT tests, verifying that they pass

The base branch was changed.