Add rabbitmq monitoring docs (#852)

Adds docs on using rabbitmq operator provided
monitoring.

docs/infrastructure-rabbitmq.md

@@ -31,3 +31,44 @@ kubectl apply -k /etc/genestack/kustomize/rabbitmq-cluster/overlay
 ``` shell
 kubectl --namespace openstack get rabbitmqclusters.rabbitmq.com -w
 ```
+
+## RabbitMQ Operator Monitoring
+
+RabbitMQ Operator provides ServiceMonitor and PodMonitor CRDs to expose scrape endpoints for rabbitmq
+cluster and operator.
+
+!!! warning
+
+    Make sure Prometheus Operator is deployed prior to running these commands. It will error out if the
+    rquired CRDs are not already installed.
+
+Check if the required CRDs are installed
+
+``` shell
+kubectl get customresourcedefinitions.apiextensions.k8s.io servicemonitors.monitoring.coreos.com
+```
+
+if the CRDs are present you can run the following
+
+```shell
+kubectl apply --filename https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/observability/prometheus/monitors/rabbitmq-servicemonitor.yml
+
+kubectl apply --filename https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/observability/prometheus/monitors/rabbitmq-cluster-operator-podmonitor.yml
+```
+
+then,
+
+```shell
+for file in $(curl -s https://api.github.com/repos/rabbitmq/cluster-operator/contents/observability/prometheus/rules/rabbitmq | jq -r '.[].download_url'); do   kubectl apply -n prometheus -f $file; done
+
+for file in $(curl -s https://api.github.com/repos/rabbitmq/cluster-operator/contents/observability/prometheus/rules/rabbitmq-per-object | jq -r '.[].download_url'); do   kubectl apply -n prometheus -f $file; done
+```
+
+In order for these to work we need to also make sure that they match the `ruleSelector` from Prometheus deploy.
+For genestack deploys run
+
+```shell
+kubectl get prometheusrule -n prometheus -o name | xargs -I {} kubectl label -n prometheus {} release=kube-prometheus-stack --overwrite
+```
+This will get all the rules in prometheus namespace and apply `release=kube-prometheus-stack` label. At this point the alerts will be configured
+in prometheus.

docs/monitoring-getting-started.md

@@ -43,10 +43,14 @@ Configure the alert manager to send the specified alerts to slack as an example,
 
 ### Update alerting rules
 
-Within the genestack repo we can update our alerting rules via the alerting_rules.yaml to fit our needs
+Within the genestack repo we can update our custom alerting rules via the alerting_rules.yaml to fit our needs
 
 View alerting_rules.yaml in:
 
 ``` shell
 less /etc/genestack/helm-configs/prometheus/alerting_rules.yaml
 ```
+
+However, many opreators comes with ServiceMonitor and PodMonitor services. These services expose, scrape endpoints
+out of the box. These operators will also provide alerting rules curated for the specific service. See specific
+service install for any monitoring rules. Example: [RabbitMQ Operator Monitoring](infrastructure-rabbitmq.md#rabbitmq-operator-monitoring)