Gitlab pipelines can become hung for a long time

[scottwittenburg]

Starting around the middle of February, 2022, we started noticing pipelines that were marked as "Running" in gitlab, but further inspection showed that while there were no failures in the pipeline, after a certain point in the stages, jobs were marked only as "Created". No running jobs could be found in these pipelines.

We discovered that if we simply clicked the cancel button on any of the "Created" jobs, this would not only result in that job being marked as "Canceled", but also it seemed to remind gitlab of the other forgotten jobs that were able to run, and it would then trigger them.

This short video clip illustrates the behavior for one such pipeline which sat marked as running for over a month.