[request]: Monitoring account support

[clalbus]

Use case. Why is this important?

In multi account setup it is suggested to use monitoring accounts for easier metric access i.e. view multi account metrics through Cloudwatch in single account.

It is possible to filter metrics by account ID using setAccountId(String accountId) on MetricDataQuery when retrieving Cloudwatch metrics from monitoring account.

However, the above applies only to users who use monitoring accounts.

WHY

We are picking up pace with multi account setup and would like to simplify Cloudwatch metric ingestion into Prometheus via monitoring account.

Not only it simplifies IAM management i.e. no more role/policy per account, but with account filtering it allows us to re-use existing Cloudwatch exporters (requiring simple config change).

How do you think the new configuration should look like?

Example:

---
region: eu-west-1
metrics:
  - aws_namespace: AWS/ELB
+   aws_account_id: 111111111111 #optional
    aws_metric_name: RequestCount
    aws_dimensions: [AvailabilityZone, LoadBalancerName]
    aws_dimension_select:
      LoadBalancerName: [myLB]
    aws_statistics: [Sum]

[matthiasr]

At a high level this makes sense. I have never worked with monitoring accounts, can you describe a little more how you would build the overall setup? How would you run the exporter (a single exporter for the whole account?) How would you collect, and at all the stages distinguish, the metrics from the different accounts? The account ID is not very memorable, how could we make it so that the metrics are labeled with something nicer when it's all done?

[korenyoni]

At a high level this makes sense. I have never worked with monitoring accounts, can you describe a little more how you would build the overall setup? How would you run the exporter (a single exporter for the whole account?) How would you collect, and at all the stages distinguish, the metrics from the different accounts? The account ID is not very memorable, how could we make it so that the metrics are labeled with something nicer when it's all done?

We have the same desired use case as @clalbus — it aligns with suggested AWS best practices for AWS Organizations. AWS CloudWatch monitoring accounts is one example of the many supported solutions that follow the delegated admin account pattern

For a cloudwatch_exporter integration with CloudWatch monitoring accounts, a high level overview of the setup could be:

1. Follow the documented AWS guide to delegate a monitoring account and sharing accounts. This essentially aggregates metrics from sharing accounts into the monitoring account.
2. Deploy cloudwatch_exporter in the monitoring account with the following proposed configuration (based off of the suggestion made by @clalbus):

---
region: eu-west-1
metrics:
  - aws_namespace: AWS/ELB
+   aws_account_id: 111111111111
+   aws_account_label: dev
    aws_metric_name: RequestCount
    aws_dimensions: [AvailabilityZone, LoadBalancerName]
    aws_dimension_select:
      LoadBalancerName: [myLB]
    aws_statistics: [Sum]
    
   - aws_namespace: AWS/ELB
+   aws_account_id: 222222222222
+   aws_account_label: qa
    aws_metric_name: RequestCount
    aws_dimensions: [AvailabilityZone, LoadBalancerName]
    aws_dimension_select:
      LoadBalancerName: [myLB]
    aws_statistics: [Sum]

The aws_account_label would add an additional aws_account label to the metrics. I thought of suggesting that cloudwatch_exporter would query AWS organizations and determine this label automatically, but it gives less control to the user, is probably a bad idea because we just want cloudwatch_exporter to do one thing and do it well, and it wouldn't work with CloudWatch sharing accounts that exist outside the AWS organization of the monitoring account.

Without this feature, you would not be able to tell which account the metrics came from.

But right now monitoring accounts are not supported at all, because we would need to add IncludeLinkedAccounts to the ListMetrics call.

One thing I have not fully considered yet is how a configuration would look like in a multi account, multi region setup, which is supported by CloudWatch. I suppose you would have a configuration per region, though admittedly that's pretty repetitive.

[matthiasr]

Sounds good, I think this is a good feature and a clean design. One thing I would add is that if aws_account_label is not specified, we use the account ID. This way the metrics are still unique, and the label setting becomes a convenience not a necessity.

[vladiceanu]

I have a similar use case where I want to collect the metrics from a different AWS Account into our main account's Prometheus without the need to deploy an additional prometheus-cloudwatch-exporter instance that would assume the cross-account IAM role. Looking forward to monitoring-account support in prometheus-cloudwatch-exporter.