Mitigation for skipped pipelines (issue #316)

[zackgalbreath]

See #316 for a description of the problem that this is mitigating

[zackgalbreath]

This approach seems to work, but it's pretty slow (~9 - 10 minutes per run).

@kotfic has some prior art that directly queries the GitLab database instead of repeatedly querying the API.

@mvandenburgh do you think that approach is still sound after our move to AWS RDS? I'm not sure how to connect to that database to test / debug a script like this.

[mvandenburgh]

@mvandenburgh do you think that approach is still sound after our move to AWS RDS? I'm not sure how to connect to that database to test / debug a script like this.

Yeah, anything like that that worked before with the in-cluster DB should still work. The RDS database lives in the same VPC as the old one, so from the perspective of the components of the cluster that directly interact with the DB, nothing has really changed, it's still just a postgres database.

[zackgalbreath]

@mvandenburgh what do you think about merging this mitigation as-is and trying to optimize its runtime later if it actually proves to be a problem?

[mvandenburgh]

Sounds good to me. Since this solution is proven to work, I agree we should merge as-is and focus on higher priority things (as long as peformance is sufficient)

[tgamblin]

Can this description be updated with rationale?