feat: add some kind of error/failure-handling ability to promotion processes

[krancour]

Discussed in #3227

[dbason]

The steps in the discussion seem reasonable. One additional ask would be the ability to run steps if an analysisTemplate fails, or have that also result in the promotion failure steps being run. The usecase for this would be setting up automated rollbacks in the event of a promotion or analysisRun failure.

Useful information to include the ctx for the promotion would be the step it failed on and the freight prior to promotion.

[krancour]

@dbason #2972 might be closer to what you're looking for.

[dbason]

Yup that works for the automated rollback. Would still like the ability to run a separate step on failure to trigger a slack webhook notification.

[jessesuen]

Copying over @krancour's proposal from the discussion from which this issue was created

Promotions stop on the first failure, so you wouldn't reach your http step, but I think your use case seems perfectly valid. I think we'd first need to introduce one or more of the following:
A step-level option that indicates the Promotion may continue even if the step fails. (Although I think most processes would then encounter additional failures on subsequent steps, this might not be as useful as it sounds at first.)
Something like "catch" to define a separate sequence of steps for when the main process fails.
Something like "finally" to define a separate sequence of steps that runs no matter what.

After some more discussion, we favor something more like GitHub Action's status check functions, which is what is being proposed in #3639.

Closing in favor of that.