JMX: Tomcat sessions is always exported as 0

[sky333999]

Component(s)

jmx-metrics

What happened?

Description

When using tomcat as the target system for jmx-metrics, tomcat.sessions is always emitted as 0 despite the MXBeans showing a non-zero value for the activeSessions attribute.

Steps to Reproduce

• Spin up a tomcat server and install any sample web app that leverages sessions on it.
• Create sessions and confirm by looking at the Tomcat manager that sessions is non-zero.

In this example, **shoppingcart** is a sample app and as can be seen, the sessions count is 1.

• Use a tool such as jconsole or jmxterm to query MXBeans and confirm activeSessions is non-zero.

$>get -b Catalina:context=/shoppingcart,host=localhost,type=Manager activeSessions
#mbean = Catalina:context=/shoppingcart,host=localhost,type=Manager:
activeSessions = 1;

The above is the output from jmxterm to query the MXBeans. As can be seen, the value matches the expectation.

• Observe the value of tomcat.sessions coming from jmx-metrics. For example, setup an OTel collector pipeline with a JMX receiver and debug exporter.

Expected Result

I expect the value of tomcat.sessions as per the definition here to also be non-zero.

Actual Result

The value of tomcat.sessions is always 0.

Component version

v1.35.0

Log output

No response

Additional context

No response

[jefchien]

Looked into the code a bit and I think I figured it out. It's because the MBean for the tomcat manager

opentelemetry-java-contrib/jmx-metrics/src/main/resources/target-systems/tomcat.groovy

Lines 18 to 19 in b7345dc

	def beantomcatmanager = otel.mbean("Catalina:type=Manager,host=localhost,context=*")
	otel.instrument(beantomcatmanager, "tomcat.sessions", "The number of active sessions.", "sessions", "activeSessions", otel.&doubleValueCallback)

and many others are being defined as a single bean

opentelemetry-java-contrib/jmx-metrics/src/main/groovy/io/opentelemetry/contrib/jmxmetrics/OtelHelper.groovy

Lines 80 to 83 in b7345dc

	MBeanHelper mbean(String objNameStr) {
	def mbeanHelper = new MBeanHelper(jmxClient, objNameStr, true)
	mbeanHelper.fetch()
	return mbeanHelper

when in reality, the object name includes a wildcard and after performing a query against the MBean server, Catalina:type=Manager,host=localhost,context=* resolves into multiple MBeans

• Catalina:context=/shoppingcart,host=localhost,type=Manager
• Catalina:context=/,host=localhost,type=Manager
• etc.

What ends up happening is that the gatherer will fetch all the MBeans that match the pattern and then because it thinks it's a single MBean, it only keeps the first one.

opentelemetry-java-contrib/jmx-metrics/src/main/groovy/io/opentelemetry/contrib/jmxmetrics/MBeanHelper.groovy

Lines 93 to 99 in b7345dc

	@PackageScope List<GroovyMBean> getMBeans() {
	if (mbeans == null || mbeans.size() == 0) {
	logger.warning("No active MBeans. Be sure to fetch() before updating any applicable instruments.")
	return []
	}
	return isSingle ? [mbeans[0]]: mbeans
	}

So, the metrics don't represent an aggregated value of all the beans that match the pattern. It's just the first one, which happens to be 0.

[jefchien]

It looks like this is a simple case of misconfiguration. The JVM metrics are defined correctly using otel.mbeans

opentelemetry-java-contrib/jmx-metrics/src/main/resources/target-systems/jvm.groovy

Line 21 in b7345dc

def garbageCollector = otel.mbeans("java.lang:type=GarbageCollector,*")

The Tomcat metrics just need to be updated to use otel.mbeans instead of otel.mbean.

[jefchien]

Not quite so simple. The gatherer needs to be updated to support aggregating metrics with the same attributes. After changing the metrics to otel.mbeans, it's calling record on each of the individual data points, which results in it dropping all but the first.

{"caller":"subprocess/subprocess.go:280","msg":"WARNING: Instrument tomcat.sessions has recorded multiple values for the same attributes: {}","kind":"receiver","name":"jmx","data_type":"metrics"}

[breedx-splk]

Just to circle back on this -- there's currently an effort underway to deprecate/remove the jmx-metrics module here in favor of a new jmx-scraper. That new module will use yaml configs instead of groovy, and will share configs with the jmx instrumentation in the agent.

In the first pass, it does look like we are continuing to support the opt-in aggregation flag (as implemented in #1366), but long term we want to report values from all the MBeans with some mbean name as an attribute/dimension so that it can be aggregated or ignored as appropriate in backends.

[SylvainJuge]

Hi,

With the jmx-scraper the metrics definitions will be in YAML, in this particular case for Tomcat metrics adding an extra metric attribute with the context allows to solve this without having to compute a sum.

In the current PR proposal #1485 (which I hope will be merged soon), the metrics for Catalina:type=Manager,host=localhost,context=* MBean definition will include the context as a metric attribute as you can see here:

opentelemetry-java-contrib/jmx-scraper/src/main/resources/tomcat.yaml

Lines 12 to 13 in 3362166

	# minor divergence from tomcat.groovy to capture metric for all deployed webapps
	context: param(context)

In the short, term once #1485 is merged and released you should be able to test this jmx-scraper component as the metrics should be the same as with the current jmx-metrics.

[breedx-splk]

Yes, I'm happy with where this has landed. The other module has the opt-in aggregation flag, and the (new) jmx scraper has the more correct dimension. Any reason to leave this open then?

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.