[GH Request] Github runners not picking up jobs

[bmtcril]

Firm Name

Axim

Urgency

High (< 3 days)

Problem/Request

Github runners seem unavailable for most jobs. Github is not reporting any issues, our org is showing 0 runners in use. This is happening in at least 3 repos, so seems org wide.

Logs for things like commitlint look like:

Requested labels: ubuntu-20.04
Job defined at: openedx/.github/.github/workflows/commitlint.yml@refs/heads/master
Reusable workflow chain:
openedx/xapi-db-load/.github/workflows/commitlint.yml@refs/pull/118/merge (d6d3143615a9b2391879f892aae3fc287030b2cd)
-> openedx/.github/.github/workflows/commitlint.yml@refs/heads/master (4cd16d7b750f2699a99dd6a85c264453d396d238)
Waiting for a runner to pick up this job...

Examples:

openedx/frontend-app-learner-portal-enterprise#1136
openedx/xapi-db-load#118
openedx/edx-platform#35206

Reasoning

Should be obvious. :) I'm looking into this currently, but could use more eyes.

[kdmccormick]

There was an incident earlier today involving degraded hosted actions runners, but it marked as resolved: https://www.githubstatus.com/incidents/y0gqch2dhw5l

Still, no running jobs: https://github.com/organizations/openedx/settings/actions/hosted-runners

[bmtcril]

I saw that, but I don't believe we're using large runners so didn't think it applied. The later redis issue also impacted actions but was supposedly fixed 5 hrs ago.

[kdmccormick]

And just like that, looks like jobs are getting processed again: https://github.com/organizations/openedx/settings/actions/hosted-runners

We've got like 14 pages of queued jobs just for edx-platform, so it'll take some time to catch up, but I think we're good now.

[github-actions]

Thank you for your report! @openedx/axim-oncall will triage within a business day. Simple requests usually take 2-3 business days to resolve; more complex requests could take longer.