[CIIP 2023] Bug: Any DAG failure between swrs_import and app_user leaves the database with no permissions

[pbastia]

There is an issue where if a DAG fails between cas_ciip_swrs_import and cas_ciip_app_user, then the DAG errors out and the permissions to the swrs schema are not restored.

One avenue to fix that would be to tell Airflow of an error case, where if one of the DAG fails then we still run the cas_ciip_app_user in something equivalent to a finally clause

Keywords: CIIP 2023 cycle

Probability (1-5): 1 (It happened once in 3 years)

Effect (1-5): 4 (new applications can't be created as the swrs data can't be imported)

Steps to reproduce the behaviour:

[pbastia]

Temporary solution: cycle the pods to restore functionality

[JoshLarouche]

Adding some framework to input severity and probability of this bug but I'm unfamiliar with the issue directly. @pbastia could you fill those out?

[pbastia]

@JoshLarouche I added some details. The likelihood of this happening is really low (airflow must stop working specifically between 2 specific tasks), and the fix is pretty easy. Still a failure point that was identified and that the team can remove, but this doesn't have to be done immediately