Remote jmx has enabled SSL. How to configure jmx-exporter for verification?

[kingEneru]

issue description

jmx_exporter able to running successful in kubernetes, but remote jmx has enabled SSL. So, when I access to jmx_exporter metrics, I met these error as below:

	at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
  at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
  at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
  ... 19 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
  javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
  at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:308)
  at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:204)
  at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:344)
  at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
  at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:134)
  ... 24 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
  at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1719)
  at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1518)
  at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
  at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
  at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:925)
  at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1295)
  at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
  at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
  at java.base/java.io.DataOutputStream.flush(DataOutputStream.java:128)
  at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:230)
  ... 28 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
  at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
  at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
  at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
  at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
  at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
  ... 36 more

configuration

• jmx_exporter image version: 0.20.0
• running mode: jmx_prometheus_httpserver
• config.yaml

jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:{{ .Values.jmxPort }}/jmxrmi
lowercaseOutputName: true
lowercaseOutputLabelNames: true
ssl: true
username: admin
password: {{ default "" .Values.jmxPassword | quote }}

[dhoard]

When running the exporter in standalone mode with JMX being protected by SSL you have to...

1. Create a trust store to be used by the exporter.

2. Added the trusted certificate to the trust store )if you are not using a certificate signed by a public certificate authority)

3. Added the following properties to your exporter command line

-javax.net.ssl.trustStore=<file> -javax.net.ssl.trustStorePassword=<trustStore password>

[kingEneru]

When running the exporter in standalone mode with JMX being protected by SSL you have to...

1. Create a trust store to be used by the exporter.
2. Added the trusted certificate to the trust store )if you are not using a certificate signed by a public certificate authority)
3. Added the following properties to your exporter command line

-javax.net.ssl.trustStore=<file> -javax.net.ssl.trustStorePassword=<trustStore password>

Thanks for your prompt reply. And do I need to add the keystore command line?

-Djavax.net.ssl.keyStore=/home/user/.keystore
-Djavax.net.ssl.keyStorePassword=changeit

[dhoard]

I don't believe the keyStore values are required.

[kingEneru]

I don't believe the keyStore values are required.

Okay，I generated truststore.p12 through tomcatJMX.cer file, and specify trustStore in commandline, but still got the above memtioned error.

• Create truststore
  keytool -import -keystore ./truststore.p12 -storepass changeit-noprompt -trustcacerts -v -alias jmxssl -file ./tomcatJMX.cer
• Running command:
  java -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit

[dhoard]

This command line is incorrect...

java -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit

This is passing -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit as arguments to the exporter.

The correct command line should be...

java -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml

---

The use of the Java agent is strongly recommended. Some JVM metrics can't be captured when running the standalone exporter.

[kingEneru]

This command line is incorrect...

java -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit

This is passing -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit as arguments to the exporter.

The correct command line should be...

java -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml

The use of the Java agent is strongly recommended. Some JVM metrics can't be captured when running the standalone exporter.

It seems that the agent mode cannot be used because I deployed the java application and jmx_exporter in a kubernetes environment.

[dhoard]

Deployment of the JMX Exporter Java agent works in Kubernetes.

[kingEneru]

Hi @dhoard , I tried again based on the correct command you provided. Unfortunately, I still got the same error. Then, I checked the SSL related source code and found that after I deleted this line and tested it in local, I was able to pass the SSL authentication successful. This Is it a bug?

[dhoard]

@kingEneru I am reviewing #947, which appears to fail a new integration test I have created (not yet merged into main.)

EDIT: The code in main works correctly.

[kingEneru]

@kingEneru I am reviewing #947, which appears to fail a new integration test I have created (not yet merged into main.)

EDIT: The code in main works correctly.

That's weird...
Does SSL authentication require that the jdk version of jmx exporter is the same as the jdk version of jmx server?

[dhoard]

That's weird... Does SSL authentication require that the jdk version of jmx exporter is the same as the jdk version of jmx server?

It does not. This is a configuration issue. I just updated PR #947 with the missing Java system property.

[dhoard]

@kingEneru When configuring RMI for SSL, the expectation is that the RMI registry is also configured for SSL. This requires the Java system property...

  -Dcom.sun.management.jmxremote.registry.ssl=true

... to be defined when launching your application.

I have merged an integration test that tests/validates RMI with SSL. @unitsvc also validated that adding the Java system property resolves the issue.

[kingEneru]

@kingEneru When configuring RMI for SSL, the expectation is that the RMI registry is also configured for SSL. This requires the Java system property...

  -Dcom.sun.management.jmxremote.registry.ssl=true

... to be defined when launching your application.

I have merged an integration test that tests/validates RMI with SSL. @unitsvc also validated that adding the Java system property resolves the issue.

Okay, It's working successfully and normally now. Thanks a lot