Demo Dashboard Application Metrics are not service specific

[flands]

Bug Report

Which version of the demo you are using? 1.10.0

Symptom

The Application metrics panels in the Grafana Demo Dashboard are not service specific like other panels on the dashboard. This is confusing and also results in missing observability

What is the expected behavior?

1. Application metrics panel is applicable to all service and updates based on the service set at the top of the dashboard
2. Application metrics widgets are service specific not language specific
3. The observability of the application can be determined from the Application metrics panel

What is the actual behavior?

1. The CPU% and Memory widgets are applicable to Python services only - not specific to the service selected at the top of the dashboard and lacking information for other languages
2. Quote service batch span processor seems irrelevant to Application metrics / health
3. Beyond Python CPU/mem and recommendation rate, no application metrics are shown (perhaps this is a feature request more than a bug)

Reproduce

Start version 1.10, open Grafana, go to Dashboard > Demo > Demo Application, and scroll down to the Application Metrics panel.

Additional Context

Beyond demonstrating the stable capabilities of OTel components, the demo application should also demonstrate that you can achieve observability. The Grafana Demo dashboard doesn't achieve this today.

Not a regression - this problem has existed since 1.7.0 and likely earlier.

[puckpuck]

The dashboard does behave as it is designed. Only some of the services emit custom application metrics, and what you see are several of those custom metrics displayed. Do note that not all custom app metrics are on that dashboard.

I want to add infra metrics, but these are collected from the orchestration/infra layer and not the application layer. We have discussed it a couple of times in SIG meetings, and though we all want infra metrics for each component, doing it in a way that is packaged within the confines of the demo has proven to be the challenge.

[julianocosta89]

Anything against adding docker_stats receiver in the otelcol-config?
That would give us a couple of container metrics when running locally.

Then in the values.yaml for helm we could add k8s related stuff.

The configs would differ from docker to k8s, but I guess this is expected anyways.

[rogercoll]

Anything against adding docker_stats receiver in the otelcol-config?

+1. I think it would make sense to create an issue just for container related metrics. I was thinking of adding the container filelog operator in the k8s deployment too.

[julianocosta89]

@rogercoll: #1650
WDYT?