Production - [Alerting] Servicing jobs in R&D queues alert

[dotnet-eng-status]

💔 Metric state changed to alerting

One or more servicing jobs were executed in a R&D queue, the expectation is that FR investigates why the jobs weren't redirected. The most common reasons are:

• The job was sent to an on-prem queue, an on-prem queue is one that has osx, arm64 or perf within the name
  • We don't have physical hardware for servicing work so on-prem queues should be excluded from this effort. To fix the alert, we need to update the query and add the queue name to the third line where list on-prem queues.
• The job was sent to a queue that doesn't have a corresponding servicing queue
  • We need to create the missing queue in helix machines repo

Next steps:

1. Go to https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/historical/backend-status?orgId=1&viewPanel=72
2. Investigate every job in the table and decide if we need to update the alert to exclude the job or if need to create a servicing queue for it

For more context go here

• ServicingJobs 3

Go to rule

@dotnet/dnceng, @dotnet/prodconsvcs, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-5aa74f27ef6445ce9d3d8d3d382e7e35

[dotnet-eng-status]

💚 Metric state changed to ok

One or more servicing jobs were executed in a R&D queue, the expectation is that FR investigates why the jobs weren't redirected. The most common reasons are:

• The job was sent to an on-prem queue, an on-prem queue is one that has osx, arm64 or perf within the name
  • We don't have physical hardware for servicing work so on-prem queues should be excluded from this effort. To fix the alert, we need to update the query and add the queue name to the third line where list on-prem queues.
• The job was sent to a queue that doesn't have a corresponding servicing queue
  • We need to create the missing queue in helix machines repo

Next steps:

1. Go to https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/historical/backend-status?orgId=1&viewPanel=72
2. Investigate every job in the table and decide if we need to update the alert to exclude the job or if need to create a servicing queue for it

For more context go here

Go to rule

[dotnet-eng-status]

💔 Metric state changed to alerting

One or more servicing jobs were executed in a R&D queue, the expectation is that FR investigates why the jobs weren't redirected. The most common reasons are:

• The job was sent to an on-prem queue, an on-prem queue is one that has osx, arm64 or perf within the name
  • We don't have physical hardware for servicing work so on-prem queues should be excluded from this effort. To fix the alert, we need to update the query and add the queue name to the third line where list on-prem queues.
• The job was sent to a queue that doesn't have a corresponding servicing queue
  • We need to create the missing queue in helix machines repo

Next steps:

1. Go to https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/historical/backend-status?orgId=1&viewPanel=72
2. Investigate every job in the table and decide if we need to update the alert to exclude the job or if need to create a servicing queue for it

For more context go here

• ServicingJobs 3

Go to rule

[dotnet-eng-status]

💚 Metric state changed to ok

One or more servicing jobs were executed in a R&D queue, the expectation is that FR investigates why the jobs weren't redirected. The most common reasons are:

• The job was sent to an on-prem queue, an on-prem queue is one that has osx, arm64 or perf within the name
  • We don't have physical hardware for servicing work so on-prem queues should be excluded from this effort. To fix the alert, we need to update the query and add the queue name to the third line where list on-prem queues.
• The job was sent to a queue that doesn't have a corresponding servicing queue
  • We need to create the missing queue in helix machines repo

Next steps:

1. Go to https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/historical/backend-status?orgId=1&viewPanel=72
2. Investigate every job in the table and decide if we need to update the alert to exclude the job or if need to create a servicing queue for it

For more context go here

Go to rule

[dotnet-eng-status]

💔 Metric state changed to alerting

One or more servicing jobs were executed in a R&D queue, the expectation is that FR investigates why the jobs weren't redirected. The most common reasons are:

• The job was sent to an on-prem queue, an on-prem queue is one that has osx, arm64 or perf within the name
  • We don't have physical hardware for servicing work so on-prem queues should be excluded from this effort. To fix the alert, we need to update the query and add the queue name to the third line where list on-prem queues.
• The job was sent to a queue that doesn't have a corresponding servicing queue
  • We need to create the missing queue in helix machines repo

Next steps:

1. Go to https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/historical/backend-status?orgId=1&viewPanel=72
2. Investigate every job in the table and decide if we need to update the alert to exclude the job or if need to create a servicing queue for it

For more context go here

• ServicingJobs 3

Go to rule

[dotnet-eng-status]

💚 Metric state changed to ok

One or more servicing jobs were executed in a R&D queue, the expectation is that FR investigates why the jobs weren't redirected. The most common reasons are:

• The job was sent to an on-prem queue, an on-prem queue is one that has osx, arm64 or perf within the name
  • We don't have physical hardware for servicing work so on-prem queues should be excluded from this effort. To fix the alert, we need to update the query and add the queue name to the third line where list on-prem queues.
• The job was sent to a queue that doesn't have a corresponding servicing queue
  • We need to create the missing queue in helix machines repo

Next steps:

1. Go to https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/historical/backend-status?orgId=1&viewPanel=72
2. Investigate every job in the table and decide if we need to update the alert to exclude the job or if need to create a servicing queue for it

For more context go here

Go to rule

[dotnet-eng-status]

💔 Metric state changed to alerting

One or more servicing jobs were executed in a R&D queue, the expectation is that FR investigates why the jobs weren't redirected. The most common reasons are:

• The job was sent to an on-prem queue, an on-prem queue is one that has osx, arm64 or perf within the name
  • We don't have physical hardware for servicing work so on-prem queues should be excluded from this effort. To fix the alert, we need to update the query and add the queue name to the third line where list on-prem queues.
• The job was sent to a queue that doesn't have a corresponding servicing queue
  • We need to create the missing queue in helix machines repo

Next steps:

1. Go to https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/historical/backend-status?orgId=1&viewPanel=72
2. Investigate every job in the table and decide if we need to update the alert to exclude the job or if need to create a servicing queue for it

For more context go here

• ServicingJobs 2

Go to rule

[ilyas1974]

I think this is a "one off" as we have not updated VS in a while. Once #4609 is merged, if we see this issue again, we can investigate further.