BOM On-call Onboarding

[robrap]

Follow instructions in https://2u-internal.atlassian.net/wiki/spaces/AT/pages/16387065/On-Call+Arch-BOM#On-call-Onboarding

• Robert
  • Update P2+ alert delay to 11am ET #942
  • Set up on-call shadowing
• Ray
  • Review docs as details in the wiki page.
  • Verify you will see "chore: Update code-owners mapping" github notifications.
  • Attend 1-hour intro meeting (scheduled Feb 24)
  • Start ~3 week (negotiable) shadowing.
• Bernie
  • Review docs as details in the wiki page.
  • Verify you will see "chore: Update code-owners mapping" github notifications.
  • Attend 1-hour intro meeting (scheduled Feb 24)
  • Start ~3 week (negotiable) shadowing.

[rayzhou-bit]

@robrap Some questions / things I noticed from reviewing the docs:

• I could not access https://edx-asgard.tools.edx.org/ (for clearing out old ASGs).
• I don't see pipeline failure tickets. Are the tickets somewhere else?
• Should we join the alerts google group?

[robrap]

@rayzhou-bit:

1. Asgard is only accessible by SRE. I'm not sure what docs of ours mention Asgard, but we should update them to make this clear.
2. We haven't been following that pipeline failure label process in a while.
   a. Once upon a time, there were lots of issues that we were able to resolve with retries, but we needed a system to capture and not duplicate these.
   b. We should decide whether we wish to re-educate ourselves with this process, or abandon it and update the docs. We could simply search for "gocd" or "pipeline" and hope that someone used one of these terms in the ticket.
3. I think that google group can be retired, but not certain. See newly created ticket: Retire the arch-bom-gocd-alerts Google Group #941.

[rayzhou-bit]

@robrap

Thanks for clearing those up! Asgard was mentioned on an SRE doc so it should be ok. It's linked from our GoCD runbook. I was going through the links to make sure I had access to what is needed in case anything went wrong.

[robrap]

@rayzhou-bit : I should also mention that Asgard is only used with EC2, so when our containerization work is complete, that will be a thing of the past.