Add support for monitors in DD metricprovider

[indeed-kelvin]

Summary

Enhance the Datadog metric provider to support using the state of a Datadog monitor as the source for analysis results, in addition to the current support for Datadog queries. If this enhancement request is accepted, we are willing to contribute to its implementation to make it happen.

Use Cases

This feature would be particularly useful in scenarios where the desired metric for analysis is already encapsulated in a Datadog monitor. Using the state of a Datadog monitor provides several advantages:

1. Reusability: Teams can leverage existing Datadog monitors without the need to rewrite or duplicate query logic specifically for Argo Rollouts.
2. Consistency: Ensures alignment between monitoring alerts in Datadog and analysis conditions in Argo Rollouts by referencing the same source of truth.
3. Ease of Use: Simplifies configuration by abstracting complex query logic into monitors, which are often easier to manage and version-control.
4. Flexibility: Allows teams to incorporate complex monitor states (e.g., composite monitors) into rollout analysis without additional setup.

Example use case: A team has a composite Datadog monitor that assesses service health across multiple dimensions, such as latency, error rates, and throughput. Instead of recreating these dimensions in Argo Rollouts, they could directly use the monitor's state to determine the rollout's health.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[zachaller]

#3892

[indeed-kelvin]

@zachaller - Based on the link you shared, am I correct in understanding that you want this functionality to be implemented as a separate analysis plugin? If we were to develop this plugin, would the y'all be open to reviewing and adopting it into the main branch?

[zachaller]

@indeed-kelvin based off the comments in that previous thread and feedback from @meeech I was not 100% sure of the use cases (I don't and never have used Datadog) and so suggested a plugin. If you want to create a plugin we can for sure put it under argoprojlabs with all the other plugins that you can then maintain etc.

As for merging the changes back into mainline I am mixed on that because as a very small maintainer group we can not become experts on all possibly providers and hence why the plugin system was create. There would have to be an overwhelming reason to add it to mainline or huge adoption of say that provider which I don't know if I see for metric providers to be honest.