Fix jmx ssl not work issue #947

unitsvc · 2024-04-24T03:18:20Z

config.yaml

startDelaySeconds: 0
username: admin
password: password
jmxUrl: "service:jmx:rmi:///jndi/rmi://127.0.0.1:1234/jmxrmi"
ssl: true
lowercaseOutputName: false
lowercaseOutputLabelNames: false

start jmx_prometheus_httpserver

java -Djavax.net.ssl.trustStore=jmx.p12 \
-Djavax.net.ssl.trustStorePassword=changeit \
-XshowSettings:vm -jar jmx_prometheus_httpserver-0.20.0.jar 49101 ./config.yaml

Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake]
        at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:138)
        at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:220)
        at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
        ... 19 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
        at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:308)
        at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:204)
        at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:344)
        at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
        at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:134)
        ... 24 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
        at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1715)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1514)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
        at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:921)
        at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1291)
        at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
        at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
        at java.base/java.io.DataOutputStream.flush(DataOutputStream.java:128)
        at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:230)
        ... 28 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
        at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
        at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
        ... 36 more

Signed-off-by: unitsvc <[email protected]>

unitsvc · 2024-04-24T04:54:40Z

@dhoard The pull request will close issue #942

unitsvc · 2024-04-24T05:15:33Z

we need open ssl, can you check this feature in jdk17.

dhoard · 2024-04-24T17:04:38Z

@unitsvc What Java version are you testing? I created an integration test using RMI and SSL (not yet merged to main.)

The current code in main works on the following Java versions, but the change in this PR causes exceptions on all tested Java versions.

adoptopenjdk/openjdk8-openj9
adoptopenjdk/openjdk11-openj9
alibabadragonwell/dragonwell:8
alibabadragonwell/dragonwell:11
alibabadragonwell/dragonwell:17
alibabadragonwell/dragonwell:21
amazoncorretto:8
amazoncorretto:11
amazoncorretto:17
amazoncorretto:21
antublue/openlogic-openjdk:8
antublue/openlogic-openjdk:11
antublue/openlogic-openjdk:17
antublue/openlogic-openjdk:21
azul/zulu-openjdk:8
azul/zulu-openjdk:11
azul/zulu-openjdk:17
azul/zulu-openjdk:21
azul/zulu-openjdk:22
azul/prime:8
azul/prime:11
azul/prime:17
bellsoft/liberica-openjdk-debian:8
bellsoft/liberica-openjdk-debian:11
bellsoft/liberica-openjdk-debian:17
bellsoft/liberica-openjdk-debian:21
bellsoft/liberica-openjdk-debian:22
bitnami/java:1.8
bitnami/java:11
bitnami/java:17
bitnami/java:21
bitnami/java:22
eclipse-temurin:8
eclipse-temurin:11
eclipse-temurin:17
eclipse-temurin:21
eclipse-temurin:22
eclipse-temurin:8-alpine
eclipse-temurin:11-alpine
eclipse-temurin:17-alpine
eclipse-temurin:21-alpine
eclipse-temurin:22-alpine
ghcr.io/graalvm/jdk:java11
ghcr.io/graalvm/jdk:java17
ghcr.io/graalvm/jdk:21
ghcr.io/graalvm/jdk:22
ibm-semeru-runtimes:open-8-jdk-focal
ibm-semeru-runtimes:open-11-jdk-focal
ibm-semeru-runtimes:open-17-jdk-focal
ibm-semeru-runtimes:open-21-jdk-focal
konajdk/konajdk:8
konajdk/konajdk:11
konajdk/konajdk:17-tlinux
mcr.microsoft.com/openjdk/jdk:8-mariner
mcr.microsoft.com/openjdk/jdk:11-mariner
mcr.microsoft.com/openjdk/jdk:17-mariner
mcr.microsoft.com/openjdk/jdk:21-mariner
mcr.microsoft.com/openjdk/jdk:11-mariner-cm1
mcr.microsoft.com/openjdk/jdk:17-mariner-cm1
mcr.microsoft.com/openjdk/jdk:21-mariner-cm1
mcr.microsoft.com/openjdk/jdk:11-ubuntu
mcr.microsoft.com/openjdk/jdk:17-ubuntu
mcr.microsoft.com/openjdk/jdk:21-ubuntu
openjdk:8
openjdk:11
openjdk:17
openjdk:21
openjdk:22
sapmachine:11
sapmachine:17
sapmachine:21
sapmachine:22

unitsvc · 2024-04-25T01:27:31Z

@dhoard Jmx ssl test jdk version:

local jdk version

java version "17.0.9" 2023-10-17 LTS
Java(TM) SE Runtime Environment (build 17.0.9+11-LTS-201)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.9+11-LTS-201, mixed mode, sharing)

k8s jdk version

openjdk version "17.0.10" 2024-01-16 LTS
OpenJDK Runtime Environment (Red_Hat-17.0.10.0.7-1) (build 17.0.10+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.10.0.7-1) (build 17.0.10+7-LTS, mixed mode, sharing)

unitsvc · 2024-04-25T01:52:40Z

#1 Create CA

keytool -genkeypair -alias jmx20240425 -keyalg RSA -keysize 2048 -validity 3650 -keystore jmx.jks -storepass changeit -keypass changeit

keytool -importkeystore -srckeystore jmx.jks -destkeystore jmx.p12 -srcstoretype jks -deststoretype pkcs12 -srcstorepass changeit -deststorepass changeit

#2 JmxSSL.java

package jmx;

import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Locale;

public class JmxSSL {
    private static final SimpleDateFormat SIMPLE_DATE_FORMAT = new SimpleDateFormat("yyyy-MM-dd | HH:mm:ss.SSS",
            Locale.getDefault());

    public static void main(String[] args) throws Exception {
        System.out.println(
                String.format(
                        "%s | %s | INFO | %s | %s",
                        SIMPLE_DATE_FORMAT.format(new Date()),
                        Thread.currentThread().getName(),
                        JmxSSL.class.getName(),
                        "Running"));

        Thread.currentThread().join(); // wait forever
    }
}

#4 start jmx server

java \
-Djavax.net.debug=all \
-Dcom.sun.management.jmxremote=true \
-Dcom.sun.management.jmxremote.port=1234 \
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.ssl=true \
-Djavax.net.ssl.keyStore=jmx.p12 \
-Djavax.net.ssl.keyStorePassword=changeit \
-Dcom.sun.management.jmxremote.password.file=./jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=./jmxremote.access \
JmxSSL.java

#5 config.yaml

startDelaySeconds: 0
username: admin
password: "123456"
jmxUrl: "service:jmx:rmi:///jndi/rmi://127.0.0.1:1234/jmxrmi"
ssl: true
lowercaseOutputName: false
lowercaseOutputLabelNames: false

#6 start jmx client

java \
-Djavax.net.debug=all \
-Djavax.net.ssl.trustStorePassword=changeit \
-Djavax.net.ssl.trustStore=jmx.p12 \
-XshowSettings:vm \
-jar ./jmx_prometheus_httpserver.jar 8081 ./config.yaml

Then visit: http://127.0.0.1:8081/metrics

@dhoard You can try it. if use main branch, it will not work, but this pr will work for ssl.

dhoard · 2024-04-25T04:03:23Z

@unitsvc When configuring RMI for SSL, the expectation is that the RMI registry is also configured for SSL.

You are missing a Java system property in step #4 when starting the application:

  -Dcom.sun.management.jmxremote.registry.ssl=true

If I remove this property, I can reproduce the Caused by: java.io.EOFException: SSL peer shut down incorrectly exception.

unitsvc · 2024-04-25T05:32:40Z

@dhoard work , thx.

java \
-Djavax.net.debug=all \
-Dcom.sun.management.jmxremote.registry.ssl=true \
-Dcom.sun.management.jmxremote=true \
-Dcom.sun.management.jmxremote.port=1234 \
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.ssl=true \
-Djavax.net.ssl.keyStore=jmx.p12 \
-Djavax.net.ssl.keyStorePassword=changeit \
-Dcom.sun.management.jmxremote.password.file=./jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=./jmxremote.access \
JmxSSL.java

dhoard · 2024-04-25T05:38:29Z

@unitsvc thanks for the confirmation.

I have merged the integration test...

https://github.com/prometheus/jmx_exporter/blob/main/integration_test_suite/integration_tests/src/test/java/io/prometheus/jmx/test/rmi/ssl/MinimalTest.java

... and associated integration test resources...

https://github.com/prometheus/jmx_exporter/tree/main/integration_test_suite/integration_tests/src/test/resources/io/prometheus/jmx/test/rmi/ssl/MinimalTest/Standalone

dhoard · 2024-04-25T05:39:02Z

Closing. Working correctly.

unitsvc · 2024-04-25T10:01:52Z

@dhoard config.yaml can you add a toggle to control this line of code, which is turned on by default because this line of code is not compatible with collected. Projects cannot be smoothly migrated.

dhoard · 2024-04-25T12:12:34Z

@unitsvc collected? ... are you referring to collectd?

My concern with allowing users to disable the RMI SSL registry requirement is that it sets up a half-secure configuration.

I can entertain adding a Java system property or environment variable to control the behavior for a specific migration scenario, but I feel we shouldn't add a proper/documented configuration value to allow a configuration scenario that is only half-secure.

unitsvc · 2024-04-25T12:36:01Z

@dhoard Yes, but the old system uses ssl, it is not allowed to disable ssl, if you add this parameter, collectd will not work, and this parameter is only valid for jdk5, adding a switch to let the user decide whether to enable or not, this does not break the original function, and greatly improve the system compatibility.

dhoard · 2024-04-25T12:58:13Z

... this parameter is only valid for jdk5

While the documentation makes this statement, testing shows it's only partially correct and may be required depending on JVM system properties.

Will adding an undocumented environment variable work for your migration?

export RMI_REGISTRY_SSL_DISABLED=true

unitsvc · 2024-04-25T13:23:08Z

It's too late today. I'll test it tomorrow.

unitsvc · 2024-04-26T01:36:29Z

@dhoard if jmx server remove -Dcom.sun.management.jmxremote.registry.ssl=true , and jmx_prometheus_httpserver add export RMI_REGISTRY_SSL_DISABLED=true, will not work.

JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake]
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
        at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
        at io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:126)
        at io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:771)
        at io.prometheus.client.Collector.collect(Collector.java:45)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:204)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:219)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:152)
        at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:71)
        at io.prometheus.client.exporter.common.TextFormat.writeFormat(TextFormat.java:53)
        at io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:100)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
        at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:852)
        at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
        at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:819)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake]
        at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:138)
        at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:220)
        at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
        ... 19 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
        at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:308)
        at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:204)
        at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:344)
        at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
        at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:134)

dhoard · 2024-04-26T10:00:25Z

@unitsvc I am proposing adding the environment variable to resolve your migration scenario.

The code has not been changed/merged into main.

If this is a valid solution for your use case, I'll make the changes and merge it into for main for the upcoming release due very shortly.

unitsvc · 2024-04-26T10:04:14Z

@dhoard That's great!

dhoard · 2024-04-27T02:01:41Z

@unitsvc I have merged the change into main to add an environment variable to disable SSL for the RMI registry, but use SSL for other RMI calls.

This change is to allow migration use cases where RMI SSL is only partially configured on an application, it is not meant for general usage.

Partial RMI SSL is considered a security hole - this configuration change may be removed in future versions.

Environment variable:

RMI_REGISTRY_SSL_DISABLED=true

kingEneru · 2024-04-28T02:03:14Z

@unitsvc I have merged the change into main to add an environment variable to disable SSL for the RMI registry, but use SSL for other RMI calls.

This change is to allow migration use cases where RMI SSL is only partially configured on an application, it is not meant for general usage.

Partial RMI SSL is considered a security hole - this configuration change may be removed in future versions.

Environment variable:
RMI_REGISTRY_SSL_DISABLED=true

That's sound great..
And will you consider releasing a new image version for this? such as 0.20.1

dhoard · 2024-04-28T02:10:16Z

The client_java project is in the process of wrapping up work on the pushgateway. Once that is complete and published, I will create a jmx_exporter beta release.

fix jmx ssl not work issue

1cf45ae

Signed-off-by: unitsvc <[email protected]>

unitsvc changed the title ~~fix jmx ssl not work issue~~ Fix jmx ssl not work issue Apr 24, 2024

dhoard mentioned this pull request Apr 24, 2024

Remote jmx has enabled SSL. How to configure jmx-exporter for verification? #942

Closed

dhoard closed this Apr 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix jmx ssl not work issue #947

Fix jmx ssl not work issue #947

unitsvc commented Apr 24, 2024

unitsvc commented Apr 24, 2024 •

edited

Loading

unitsvc commented Apr 24, 2024

dhoard commented Apr 24, 2024

unitsvc commented Apr 25, 2024

unitsvc commented Apr 25, 2024 •

edited

Loading

dhoard commented Apr 25, 2024

unitsvc commented Apr 25, 2024

dhoard commented Apr 25, 2024

dhoard commented Apr 25, 2024

unitsvc commented Apr 25, 2024

dhoard commented Apr 25, 2024

unitsvc commented Apr 25, 2024 •

edited

Loading

dhoard commented Apr 25, 2024

unitsvc commented Apr 25, 2024

unitsvc commented Apr 26, 2024

dhoard commented Apr 26, 2024

unitsvc commented Apr 26, 2024

dhoard commented Apr 27, 2024

kingEneru commented Apr 28, 2024 •

edited

Loading

dhoard commented Apr 28, 2024

Fix jmx ssl not work issue #947

Fix jmx ssl not work issue #947

Conversation

unitsvc commented Apr 24, 2024

unitsvc commented Apr 24, 2024 • edited Loading

unitsvc commented Apr 24, 2024

dhoard commented Apr 24, 2024

unitsvc commented Apr 25, 2024

unitsvc commented Apr 25, 2024 • edited Loading

dhoard commented Apr 25, 2024

unitsvc commented Apr 25, 2024

dhoard commented Apr 25, 2024

dhoard commented Apr 25, 2024

unitsvc commented Apr 25, 2024

dhoard commented Apr 25, 2024

unitsvc commented Apr 25, 2024 • edited Loading

dhoard commented Apr 25, 2024

unitsvc commented Apr 25, 2024

unitsvc commented Apr 26, 2024

dhoard commented Apr 26, 2024

unitsvc commented Apr 26, 2024

dhoard commented Apr 27, 2024

kingEneru commented Apr 28, 2024 • edited Loading

dhoard commented Apr 28, 2024

unitsvc commented Apr 24, 2024 •

edited

Loading

unitsvc commented Apr 25, 2024 •

edited

Loading

unitsvc commented Apr 25, 2024 •

edited

Loading

kingEneru commented Apr 28, 2024 •

edited

Loading