Track status state per build step instead of per pipeline

[yasunariw]

Description of the task

Because two builds on the same pipeline + branch don't necessarily always run the same build steps, the current state tracking is insufficient for handling those cases. Instead of tracking build state per pipeline, we should track it per build step.

Some details regarding Buildkite notification behavior:

• There are two types to consider -- notifications for overall builds ("Build test: updated slack output to be consistent with Yojson pretty print #123 failed") vs those for individual steps within a larger build ("step XYZ of build test: updated slack output to be consistent with Yojson pretty print #123 failed").
• The only way to differentiate b/w them is to look at the context field, which is either buildkite/pipeline-name for the former or buildkite/pipeline-name/build-step for the latter.
• The final notification of the overall build is always sent after the final notification for any of the build steps.

The implementation:

For each status notification that passes the rule check (which should only be overall build notifications), we:

1. Retrieve all status notifications associated with this commit using GH's API
2. Filter the list to only get the status notifications that belong to the same build (in case of rebuilds)
3. Filter the list again for the most recent status notification of each build step (which should either be success or failure)
4. Update the runtime state, which maps pipeline/build step names to per-branch build states.
5. Just like before, this is then queried whenever there is an "allow_once" match. An "allow_once" match will generate a notification if any of the build steps associated with the current build have a different status value from the previous build.

Future work:

It would be nice if build failure notifications also told us which build step failed. This can be done easily by making Action.partition_status returned the build steps that failed, so that Slack.generate_status_notification can include them in the final message.

Final note:

Large diff size is due to the addition of HTTP request/response stubs - apologies.

How to test

Existing tests should pass.

Two cases are added; the second handles the behavior observed in production that motivated this PR.

status.success_test_different_steps_from_prev

This is for an incoming successful build on develop branch with a single build step, "notabot-test/build". There exist past successes for same branch + different steps, and for different branch + same step, but not for same branch + same step. Thus, it should generate a notification.

status.success_test_not_affected_by_unrelated_success_with_different_steps

This is for an incoming successful build on develop branch with two steps, "notabot-test/{build-infra,setup}". Here, previously a build with steps "build-infra" and "setup" failed on step "build-infra". Then a subsequent build with step "build" succeeded, so the overall pipeline state is a success. However, the more recent successful but unrelated build should not affect generation of another success notification, given the change in status state for "build-infra".

make test

References

• existing issue: show which build step failed #80
• Slack discussion: https://ahrefs.slack.com/archives/CKZANG2TE/p1609971822011500
• other?

[Khady]

I wonder what would be the implication of this one now that we have direct messages

[Khady]

@thatportugueseguy can we close this PR now?

[thatportugueseguy]

yes 👌