RFC Bot-supported triage

[Cheddam]

As we've been working on initiatives to improve our triage processes (like introducing a range of canned scenarios and PR checklist in #9660), it's been noted that bots would be well-suited to some of the manual work involved.

Possible tasks bots could accomplish:

• Respond immediately to new PRs, providing the PR checklist for the author to run through before a maintainer starts their review
• Label new PRs based on their target branch / contents (e.g. affects/v4, change/minor for a PR targeting the 4 branch)
• Remind authors about outstanding feedback on their PRs, and automatically close them after the stale PR window closes (this would rely on feedback-required/author being applied to avoid closing PRs that are awaiting review)
• Review failing Travis builds on PRs and suggest common resolutions (rebuild JS, fix linting, etc.)

There are a range of bot platforms available. One of the most commonly used ones seems to be Probot, and there's a small selection of official bot recipes already hosted and available for it. We could also go down the path of producing our own Probot-based bot, as it has a pretty simple API, but we'd need to look into hosting this ourselves, likely on Vercel or something similar. Alternatively, automations could be implemented directly via GitHub Actions (which supports scheduled actions, in addition to event-driven ones).

This RFC doesn't yet have a clear plan of action, so it's not ready for a vote, but I'd like to get the conversation started to see what ideas others have and to see if anyone has recommendations based on experiences implementing bots in the past.

[robbieaverill]

change/minor for a PR targeting the 4 branch

It's important that this isn't assumed based on the target branch, but from the semver definition of the changes in the PR.

Here's a relevant conversation that's been had on this topic which is worth mentioning here: #8343 (comment)

[sminnee]

This seems like a good idea.

My main suggestion would be about the process of introducing these.

IMO it would be beat that we do these incrementally so that we get a sense of how they improve flow (or don't) in the real world. A "put all possible bots in" epic is more likely to lead to bot overload, compared to "one new bot a [week/fortnight/month]".

Maybe an explicit "did this bot make maintenance better/worse/the same" question a few weeks after its gone in as well. If it hasn't really improved things it might be worth removing or reconfiguring a given bot, to keep things simple.

It would also be important to canvas both people doing maintenance work on the CMS Squad, and the community core committers, as both do important maintenance work but their context is quite different, so what is helpful for one group may not be for others.

All in all, though, I expect that some judicious introduction of bots could make things a lot more straightforward!

[chillu]

Good stuff. Agree with incremental introduction, but I'd like to see a strategy for rolling this out across ~100 repos as well. Taking probot/stale as an example, what happens if we decide later that closing after 14 days stale is too low. Do we send 100 pull requests for .github/stale.yml?

Even if this creates more work than we cumulatively save through those measures, it might still be worth it in terms of reducing context switching for maintainers, as well as encouraging contributions through faster (automated) responses. But at our scale, I think rollout needs to be considered upfront.

Also, a bit of research into what permissions we'd give to Github Apps or Github Actions from third parties. I'm assuming some of them will need write access to codebases, which significantly increases security concerns. For example, a compromised Github App could add XSS to a JS bundle in silverstripe/admin, tag a new release, and before we'll notice it might have already been installed and deployed by various projects.

[GuySartorelli]

Duplicate of the up-to-date silverstripe/.github#280