JMX Exporter Container Fails to Connect to Rest-Server JMX Port (Connection Refused)

[abdelrahman-a-ibrahim]

Description: I am using the following helm chart to deploy Kafka rest proxy with the kafka-rest image tag: 7.0.1. We are encountering an issue with internal communication between two containers in our Kubernetes deployment: the kafka-rest-server-container and the jmx-exporter-container. The kafka-rest-server-container is exposing a JMX port, configured through an environment variable, for the jmx-exporter-container to scrape metrics. However, whenever the jmx-exporter-container attempts to connect to the JMX port on localhost, it encounters a "Connection Refused" error.

Logs: Below are the logs from the jmx-exporter-container that indicate the connection refusal:

SEVERE: [!dt dt.entity.host=HOST-BF4FF85A33F813E3,dt.entity.process_group=PROCESS_GROUP-E763B61EAEB2F724,dt.entity.process_group_instance=PROCESS_GROUP_INSTANCE-9C7A9A7773EF297F,dt.kubernetes.cluster.id=e3e50ceb-9180-4ac5-937e-e87169bb2b16,dt.kubernetes.workload.kind=Deployment,dt.kubernetes.workload.name=kafka-dev-shared-services-kafka-staging-cp-kafka-rest-proxy,k8s.namespace.name=kafka,k8s.pod.name=kafka-dev-shared-services-kafka-staging-cp-kafka-rest-proxcmhh6,k8s.pod.uid=a7b3a612-4dd5-4ff3-8afe-2fa8267f482b] JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
        java.net.ConnectException: Connection refused (Connection refused)]
        ...
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
        java.net.ConnectException: Connection refused (Connection refused)

Expected Behavior: The jmx-exporter-container should successfully connect to the JMX port on the kafka-rest-server-container to retrieve metrics data.

Current Behavior: The connection to the JMX port on localhost is being refused, as shown in the logs above.

Environment:

• Kubernetes Deployment
• Rest-Server Container JMX Port: Configured via KAFKAREST_JMX_PORT environment variable
• JMX Exporter Image: solsson/kafka-prometheus-jmx-exporter@sha256:6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
• JMX Exporter Port: 5556

Relevant Configuration:

JMX Configuration:

jmx:
    enabled: true
    image: solsson/kafka-prometheus-jmx-exporter@sha256
    imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
    imagePullPolicy: IfNotPresent
    port: 5556

Kubernetes Deployment Environment Variables for the kafka-rest-server-container:

Note: tried to use JMX_PORT and got same issue !

- name: KAFKAREST_JMX_PORT
  value: "5555"

[dhoard]

@abdelrahman-a-ibrahim a few comments...

1. solsson/kafka-prometheus-jmx-exporter is extremely old. (Last updated 4 years ago)

• https://hub.docker.com/r/solsson/kafka-prometheus-jmx-exporter

2. Confluent Platform Helm Charts have been deprecated and should not be used.

• https://github.com/confluentinc/cp-helm-charts

3. The Confluent For Kubernetes Operator should be used to run/manage the Confluent REST Proxy.

• https://docs.confluent.io/operator/2.9/overview.html

4. Confluent provides examples of using using the Prometheus JMX Exporter with the Confluent REST Proxy

• https://github.com/confluentinc/jmx-monitoring-stacks