Proposal: attributes as metrics & extra metrics

[karina-calma]

Hi all,

I am reaching out with a proposal for two different extensions of your project.

In the company that I work for, we needed to extend your project in order to collect additional data from non-standard mbeans. I am opening this discussion to check if you would be interested in merging these extensions in your project.

1. Exposing string-type mbean attributes as labels.

The mbeans that we are working with have a lot of string-type attributes which are not converted into metrics by your project.

However, we need to expose the information offered by these string attributes to Prometheus, such that they can be used in queries, or be displayed in graphs.

The best way to do this (that we were able to find) is to expose these attributes as labels on the other metrics. We made this behavior configurable in the rules section of the yaml configuration file. Here is an example:

rules:
  - pattern: '{some mbean pattern}'
     name: some_metric
     attributesAsLabels:
        - firstStringAttribute
        - secondStringAttribute

This produces a metric that looks like this:

some_metric_attribute{firstStringAttribute="something",secondStringAttribute="something else"} 123

2. Adding additional metrics to an mbean.

Some of the mbeans that we are working with do not expose any useful metrics, so we implemented a way to define a custom metric through the configuration file. This is also configurable in the rules section of the yaml configuration file. Here is an example:

  - pattern: '{some mbean pattern}'
     name: some_metric
     extraMetrics:
        - name: someName
          value: someValue
          description: someDescription

This produces a metric that looks like this:

some_metric_someName{...} someValue

The description is used in the "HELP" section of the logs and indicates what purpose the metric serves.

Please let me know if you are interested in these features. If you are, I will open separate pull requests.
Also, please tell me if there is any way to achieve this behavior in another way that is already available in your project.

[dhoard]

@karina-ciupa ... just conversation...

For 1)

You should be able to expose String attribute values as labels using something like...

https://www.robustperception.io/exposing-version-numbers-with-the-jmx-exporter/

... but since the value is always a hardcoded number, I'm unsure of the actual value in dashboards/queries.

For 2)

Adding a metric that is hard coded (name, labels, value) doesn't seem useful from a metrics perspective. Can you elaborate on the use case?

[karina-calma]

Hi @dhoard,

Thank you for your quick reply.

For 1)
We need the string attributes to become labels on the metric of a different attribute. We have mbeans that have attributes such as: duration=10000, someName=someValue and someOtherName=someOtherValue. We would need to have a metric that looks like this:

some_metric_duration{someName="someValue", someOtherName="someOtherValue"} 10000

This let us draw nice graphs and implement filters based on someName and someOtherName.
Unfortunately, the solution from the article you linked does not fulfill our needs.

---

For 2)
The problem is that some of our mbeans only have string attributes, because they are only used to indicate that a given service is running. We want to generate a new metric such as "isRunning" with value 1.0, which has all the string attributes as labels.
For example our mbean has someName=someValue and someOtherName=someOtherValue as attributes and we want to produces a metric such as:

some_metric_isRunning{someName="someValue", someOtherName="someOtherValue"} 1.0

[dhoard]

@karina-ciupa Can you provide a few concrete examples of hardcoded label values?

[karina-calma]

For 1)
some_metric_duration{serviceName="my service name", userName="ADMIN"} 10000

For 2)
some_metric_isRunning{serviceName="my service name", userName="ADMIN"} 1.0

[MaBiConti]

@dhoard :
I'm chiming in, because I've been working on this feature together with @karina-ciupa .

For use-case 1), we are not hard-coding any values.

For 2), the application we want to monitor exposes MBeans that report something like:

1. MBean "Job":

   • username: "JonDoe"
   • jobname: "runReport"

2. MBean "Job":

   • username: "DonaldDuck"
   • jobname: "runReport"

3. MBean "Job":

   • username: "DonaldDuck"
   • jobname: "exportIssues"

This data is currently not extractable by JMXExporter.
We want to expose it as:

job{username="JonDoe", jobname="runReport"} 1.0
job{username="DonaldDuck", jobname="runReport"} 1.0
job{username="DonaldDuck", jobname="exportIssues"} 1.0

With that data in Prometheus, we could chart the currently running jobs by summing up the jobs, or have a filter checking how many jobs of type "runReport" are running, etc.

[fstab]

Just FYI we have our Prometheus client_java community call today, so if you happen to be available and want to talk to Doug and me about this feel free to join https://calendar.google.com/calendar/u/0/embed?src=prometheus.io_bdf9qgm081nrd0fe32g3olsld0%40group.calendar.google.com.

[dhoard]

@karina-ciupa @MaBiConti if I understand correctly...

For use case 1

metric.x label1="<MBean 1 value>" label2="<MBean 2 value>" <MBean 3 value>

This is complex since you are trying to create composite metrics based on multiple MBeans values. While I don't know your specific approach, this sounds like a collect and process scenario that could cause memory/performance issues.

For use case 2

This should already be doable as documented at https://www.robustperception.io/exposing-version-numbers-with-the-jmx-exporter/