Skip to content

Commit

Permalink
oncall docs refactor (#3757)
Browse files Browse the repository at this point in the history
# What this PR does

- Restructure the OnCall documentation to align with Grafana Labs
[documentation structure
guidance](https://grafana.com/docs/writers-toolkit/structure/).

- Updates documentation front matter to include all required fields.

## Which issue(s) this PR fixes
Related to issue #3700 

## Checklist

- [ ] Unit, integration, and e2e (if applicable) tests updated
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)

---------

Signed-off-by: Jack Baldry <[email protected]>
Co-authored-by: Jack Baldry <[email protected]>
  • Loading branch information
alyssawada and jdbaldry authored Feb 8, 2024
1 parent dd73e58 commit 80f9329
Show file tree
Hide file tree
Showing 61 changed files with 937 additions and 635 deletions.
13 changes: 11 additions & 2 deletions docs/make-docs
Original file line number Diff line number Diff line change
Expand Up @@ -689,6 +689,13 @@ POSIX_HERESTRING

case "${image}" in
'grafana/doc-validator')
if ! command -v jq >/dev/null 2>&1; then
errr '`jq` must be installed for the `doc-validator` target to work.'
note 'To install `jq`, refer to https://jqlang.github.io/jq/download/,'

exit 1
fi

proj="$(new_proj "$1")"
echo
"${PODMAN}" run \
Expand All @@ -701,8 +708,10 @@ case "${image}" in
"${DOCS_IMAGE}" \
"--include=${DOC_VALIDATOR_INCLUDE}" \
"--skip-checks=${DOC_VALIDATOR_SKIP_CHECKS}" \
/hugo/content/docs \
"$(proj_canonical "${proj}")" | sed "s#$(proj_dst "${proj}")#sources#"
"/hugo/content$(proj_canonical "${proj}")" \
"$(proj_canonical "${proj}")" \
| sed "s#$(proj_dst "${proj}")#sources#" \
| jq -r '"ERROR: \(.location.path):\(.location.range.start.line // 1):\(.location.range.start.column // 1): \(.message)" + if .suggestions[0].text then "\nSuggestion: \(.suggestions[0].text)" else "" end'
;;
'grafana/vale')
proj="$(new_proj "$1")"
Expand Down
97 changes: 70 additions & 27 deletions docs/sources/_index.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,81 @@
---
canonical: https://grafana.com/docs/oncall/latest/
title: Grafana OnCall
menuTitle: OnCall
description: Learn about the key features of Grafana OnCall and how improve your IRM solution
weight: 500

# This is the oncall index document
# Please do not make changes to the weight of this document
# The weight is set for ordering in the docs/grafana-cloud/alerting-and-irm/ folder

keywords:
- OnCall
- Grafana Cloud
- Alerts
- Notifications
- on-call
- amixr
- OnCall
- irm
title: Grafana OnCall
menuTitle: OnCall
weight: 500
- On-call
- Escalation
- IRM
canonical: https://grafana.com/docs/oncall/latest/
aliases:
- /docs/grafana-cloud/alerting-and-irm/oncall/
---

# Grafana OnCall documentation

<img src="/static/img/docs/oncall/oncall-logo.png" class="no-shadow" width="700px">

Grafana OnCall is an open source incident response management tool built to help teams improve their collaboration and
resolve incidents faster. Some of the core strengths of Grafana OnCall include:

- **Support for a broad set of monitoring systems:** Grafana OnCall supports integrations with many monitoring systems,
including Grafana, Prometheus, Alertmanager, Zabbix, and more.
- **Reduce alert noise:** Automatic alert grouping helps avoid alert storms and reduce noise during incidents.
Auto-resolve settings can resolve without human intervention when the resolve conditions are met, enabling you to
control alert noise and reduce alert fatigue.
- **Automatic escalation to on-call rotations:** Grafana OnCall’s flexible calendar integration allows you to define
your on-call rotations while managing on-call schedules in your preferred calendar application with iCal format.
Configurable alert escalation automatically escalates alerts to on-call team members, notifies slack channels, and more.
- **ChatOps focused:** Grafana OnCall integrates closely with your slack workspace to deliver alert notifications to
individuals and groups, making daily alerts more visible and easier to manage.
- **Highly customizable**: With customizable alert grouping and routing, you can decide which alerts you want to be
notified of and how, ensuring the right people are notified for the right issues.
- **Massive scalability:** Grafana OnCall is equipped with a full API and Terraform capabilities. Ready for GitOps
and large organization configuration.
Grafana OnCall is an incident response and on-call management system that helps teams reduce the stress and maintenance of being on-call. Based on the Grafana
OnCall OSS project, Grafana OnCall is available on Grafana Cloud as part of the Grafana Incident Response & Management (IRM) solution.

## What is Grafana OnCall?

Grafana OnCall is a robust on-call management and incident response tool that is conveniently integrated into your Grafana Cloud environment.

Positioned at the core of Grafana’s Incident Response & Management (IRM) solution, Grafana OnCall automatically routes and escalates alerts to on-call teams and
channels based on your predefined escalation policies and on-call schedules.

## Key features

### Diverse monitoring system support

Grafana OnCall integrates with a diverse set of monitoring systems, including Grafana, Prometheus, Alertmanager, Zabbix, and more. This flexibility ensures
that, regardless of your existing monitoring infrastructure, your team benefits from Grafana OnCall.

### Automatic escalation to on-call rotations

Efficiently manage on-call rotations using Grafana OnCall's flexible calendar options. Define on-call schedules directly in the OnCall app, in your preferred
calendar application with iCal format, or leverage Terraform within your "as-code" workflow. Configurable alert escalation routes notifications to on-call team
members, Slack channels, and other designated points, ensuring timely responses.

### ChatOps focused

Grafana OnCall integrates closely with your Slack workspace to deliver alert notifications to individuals and groups, making daily alerts more visible and
easier to manage.

### Mobile app

Access on-call alerts on the go with the dedicated mobile app, putting critical notifications in the palm of your hand.

### As-code and GitOps

Equipped with a full API and Terraform capabilities, Grafana OnCall is ready for GitOps and large organization configuration.

### Fully customizable

With customizable alert grouping and routing, you can decide which alerts you want to be notified of and how, ensuring the right people are notified for the
right issues.

## Common on-call challenges

Explore how Grafana OnCall addresses common on-call challenges:

- **Alert Noise and Fatigue:** Automatic grouping and configurable auto-resolve settings control alert noise and reduce fatigue during incidents.
- **Balancing On-Call Load:** Schedule balance feature identifies potential workload imbalances, ensuring fair distribution.
- **Tool Sprawl:** Grafana OnCall, part of the Grafana Cloud suite, centralizes alert responses and investigations.
- **Calendar Maintenance:** Manage on-call rotations efficiently with flexible calendar integration for easy scheduling and alert escalation.
- **Custom Workflows:** Tailor incident response workflows with highly customizable alert grouping and routing for targeted notifications.

## Get started

To learn more about what Grafana OnCall can do for you, explore the following topics:

{{< section >}}
53 changes: 53 additions & 0 deletions docs/sources/configure/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
title: Configure Grafana OnCall
menuTitle: Configure
description: Learn about the configuration option available for OnCall
weight: 400
keywords:
- OnCall
- Configuration
- Integration
- Escalation
- Alert templates
- Webhooks
canonical: https://grafana.com/docs/oncall/latest/configure/
aliases:
- /docs/grafana-cloud/alerting-and-irm/oncall/configure/
---

# Configure Grafana OnCall

Grafana OnCall provides a unified platform for alert routing and on-call management. This section covers the high-level technical configuration options for
Grafana OnCall so you can tailor the system to your organization's specific needs.

## Key configuration

Explore key settings and integrations that allow you to customize your incident management process.

### Configure Integrations

Integrate with your tools and alert sources to begin routing alerts with Grafana OnCall.

For detailed information on configuring integrations, refer to the [Integrations] documentation.

### Configure Escalation chains and routes

Escalation chains allow you to customize alert routing to align with your team's processes and workflows.
You have the flexibility to define how and when to escalate to different teams for different alerts.

For information on configuration options, refer to [Escalation chains and routes].

### Customize alert templates

For detailed information on customizing alert templates, refer to the [Jinja2 templating] documentation.

{{% docs/reference %}}
[Integrations]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/integrations"
[Integrations]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations"

[Escalation chains and routes]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/configure/escalation-chains-and-routes"
[Escalation chains and routes]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/escalation-chains-and-routes"

[Jinja2 templating]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/configure/jinja2-templating"
[Jinja2 templating]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/jinja2-templating"
{{% /docs/reference %}}
Original file line number Diff line number Diff line change
@@ -1,7 +1,21 @@
---
canonical: https://grafana.com/docs/oncall/latest/escalation-chains-and-routes/
title: Escalation Chains and Routes
weight: 600
title: Escalation chains and routes
menuTitle: Escalation chains and routes
description: Understand how to configure escalation chains and routes for OnCall.
weight: 300
keywords:
- OnCall
- Configuration
- Routes
- Escalation
- Alert templates
- Routing template
- Notify
canonical: https://grafana.com/docs/oncall/latest/configure/escalation-chains-and-routes/
aliases:
- /docs/grafana-cloud/alerting-and-irm/oncall/configure/escalation-chains-and-routes/
- /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes/
- ../escalation-chains-and-routes/ # /docs/oncall/<ONCALL_VERSION>/escalation-chains-and-routes/
---

# Escalation Chains and Routes
Expand All @@ -11,7 +25,7 @@ Often alerts from monitoring systems need to be sent to different escalation cha
## Routes

Routes are used to determine which escalation chain should be used for a specific alert
group. A route's ["Routing Templates"][routing-template]
group. A route's _[Routing Templates]_
are evaluated for each alert and **the first matching route** is used to determine the
escalation chain and chatops channels.

Expand All @@ -24,15 +38,16 @@ escalation chain and chatops channels.
### Manage routes

1. Open Integration page
2. Click **Add route** button to create a new route
3. Click **Edit** button to edit `Routing Template`. The routing template must evaluate to `True` for it to apply
4. Select channels in **Publish to Chatops** section
> **Note:** If **Publish to Chatops** section does not exist, connect Chatops integrations first, see more in [docs][notify]
5. Select **Escalation Chain** from the list
6. If **Escalation Chain** does not exist, click **Add new escalation chain** button to create a new one, it will open in a new tab.
7. Once created, **Reload list**, and select the new escalation chain
8. Click **Arrow Up** and **Arrow Down** on the right to change the order of routes
9. Click **Three dots** and **Delete Route** to delete the route
1. Click **Add route** button to create a new route
1. Click **Edit** button to edit `Routing Template`. The routing template must evaluate to `True` for it to apply
1. Select channels in **Publish to Chatops** section
> **Note:** If the **Publish to Chatops** section doesn't exist, connect Chatops integrations first.
> For more information, refer to [Notify people].
1. Select **Escalation Chain** from the list
1. If **Escalation Chain** does not exist, click **Add new escalation chain** button to create a new one, it will open in a new tab.
1. Once created, **Reload list**, and select the new escalation chain
1. Click **Arrow Up** and **Arrow Down** on the right to change the order of routes
1. Click **Three dots** and **Delete Route** to delete the route

### Routing based on labels

Expand All @@ -56,7 +71,7 @@ Users can create escalation chains to configure different type of escalation wor
For example, you can create a chain that will notify on-call users with high priority, and
another chain that will only send a message into a Slack channel.

Escalation chains determine Who and When to notify. [How to notify][notify] is set by the user, based on their own preferences.
Escalation chains determine Who and When to notify. How to notify is set by the user, based on their own preferences.

### Types of escalation steps

Expand All @@ -69,7 +84,7 @@ from an on-call schedule.
`Resolved automatically`.
* `Notify whole slack channel` - send a notification to the users in the slack channel.
* `Notify Slack User Group` - send a notification to each member of a slack user group.
* `Trigger outgoing webhook` - trigger an [outgoing webhook][outgoing-webhooks].
* `Trigger outgoing webhook` - trigger an [outgoing webhook].
* `Notify users one by one (round robin)` - each notification will be sent to a group of
users one by one, in sequential order in [round robin fashion](https://en.wikipedia.org/wiki/Round-robin_item_allocation).
* `Continue escalation if current time is in range` - continue escalation only if current
Expand All @@ -85,7 +100,7 @@ should have linked their Slack and OnCall users). In both cases, the filtered us
notified following their respective notification policies. However, to avoid **spamming** the Slack channel/thread,
users **won't be notified** in the alert group Slack **thread** (this is how the feature is currently implemented)
but instead notify them using their **other defined** options in
their [respective policies]([url](https://grafana.com/docs/oncall/latest/notify/#configure-user-notification-policies)).
their respective policies.

### Notification types

Expand All @@ -99,8 +114,7 @@ User can configure two types of personal notification chains:
* **Important Notifications**

In the escalation step, user can select which type of notification to use.

Check more information on [Personal Notification Preferences][notify] page.
For more information, refer to [Notify people].

### Manage Escalation Chains

Expand All @@ -117,12 +131,12 @@ Check more information on [Personal Notification Preferences][notify] page.
affect all linked Integrations and Routes.

{{% docs/reference %}}
[notify]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/notify"
[notify]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/notify"
[Notify people]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/manage/notify"
[Notify people]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/manage/notify"

[outgoing-webhooks]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/outgoing-webhooks"
[outgoing-webhooks]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/outgoing-webhooks"
[outgoing webhook]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/configure/outgoing-webhooks"
[outgoing webhook]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/outgoing-webhooks"

[routing-template]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/jinja2-templating#routing-template"
[routing-template]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/jinja2-templating#routing-template"
[Routing Templates]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/configure/jinja2-templating#routing-template"
[Routing Templates]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/jinja2-templating#routing-template"
{{% /docs/reference %}}
Original file line number Diff line number Diff line change
@@ -1,9 +1,23 @@
---
canonical: https://grafana.com/docs/oncall/latest/jinja2-templating/
title: Jinja2 templating
weight: 1000
menuTitle: Jinja2 templating
description: Understand how to configure and customize alert templates for OnCall.
weight: 700
keywords:
- OnCall
- Configuration
- Webhooks
- JSON
- Alert payload
- Conditions
canonical: https://grafana.com/docs/oncall/latest/configure/jinja2-templating/
aliases:
- /docs/grafana-cloud/alerting-and-irm/oncall/configure/jinja2-templating/
- /docs/grafana-cloud/alerting-and-irm/oncall/jinja2-templating/
- ../jinja2-templating/ # /docs/oncall/<ONCALL_VERSION>/jinja2-templating/
---


## Jinja2 templating

Grafana OnCall can integrate with any monitoring system that can send alerts via
Expand Down Expand Up @@ -75,17 +89,6 @@ not in the Grafana OnCall integrations list, you can create a generic `webhook`
integration, send an alert, and configure
your templates.

### Labels

> **Note:** Labels are currently available only in cloud.
In addition, there is a `labels` variable available to your templates, which contains all of the labels assigned
to the Alert Group, as a `dict`. This allows you to template based on labels (or a mix of labels and/or payload based data):

> **Example:**
>
> - `{{ labels.foo or payload.bar }}`
## Types of templates

Alert templates allow you to format any alert fields recognized by Grafana OnCall. You can
Expand All @@ -99,8 +102,6 @@ customization, use Jinja templates.

> **Note:** For conditional templates, the output should be `True` to be applied, for example `{{ True if payload.state == 'OK' else False }}`
See more details in the [Routes][routes] section.

#### Appearance templates

How alerts are displayed in the UI, messengers, and notifications
Expand Down Expand Up @@ -226,10 +227,3 @@ Built-in functions:
- Usage example: `{{ payload.ruleName | regex_match(".*") }}`
- `b64decode` - performs a base64 string decode
- Usage example: `{{ payload.data | b64decode }}`

{{< section >}}

{{% docs/reference %}}
[routes]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/escalation-chains-and-routes#routes"
[routes]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes#routes"
{{% /docs/reference %}}
Loading

1 comment on commit 80f9329

@MoBattah
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a comment on the exact line change which broke the link on the Inbound Email Integration docs page.

Please sign in to comment.