{TA} Improving our flake definition

[rohan-at-sentry]

Improving our flake definition (3)

• Our heuristic currently assumes that for a commit to exist on main, it must've passed CI. So when we look up test results for a commit, and we find test failures, that is a signal that the test is potentially flaky. So essentially if a commit on main has failing tests, we assume that there was a pass and then a fail on main, so the test is flaky.

• This could be better.. WE can operate in terms of code revs to better correlate commits on main to commits on the feature branch, group those commits together and actually look up test results for those commits, and read the pass/fail history to determine flakyness, rather than inferring.

  • This changes how we model Git
  • This also risks being rate limited (we could fall back)