Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple JVMs per machine with mutliple Ports #627

Closed
scMarkus opened this issue Aug 14, 2021 · 7 comments · May be fixed by #805
Closed

Multiple JVMs per machine with mutliple Ports #627

scMarkus opened this issue Aug 14, 2021 · 7 comments · May be fixed by #805

Comments

@scMarkus
Copy link

Hi

I am intending to monitor Apache Spark Executor processes with the jmx_exporter. It turns out that in our setup where we configure a certain executor size in terms of CPU and memory and nodes in the cluster can have multiple of those the start of the JVM process with the agent attached failed. This is due to the static config which tries to reuse the same port over and over again on the the physical node (Kubernetes pod in our case).

Log output when starting minimal docker-compose example:
worker1 | 21/08/14 12:31:52 INFO ExecutorRunner: Launch command: "/usr/local/openjdk-15/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=43441" "-javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.16.1.jar=12345:/opt/prometheus/config.yaml" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://[email protected]:43441" "--executor-id" "1" "--hostname" "172.100.100.3" "--cores" "1" "--app-id" "app-20210814123152-0000" "--worker-url" "spark://[email protected]:36191" worker1 | 21/08/14 12:31:52 INFO ExecutorRunner: Launch command: "/usr/local/openjdk-15/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=43441" "-javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.16.1.jar=12345:/opt/prometheus/config.yaml" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://[email protected]:43441" "--executor-id" "0" "--hostname" "172.100.100.3" "--cores" "1" "--app-id" "app-20210814123152-0000" "--worker-url" "spark://[email protected]:36191" worker1 | 21/08/14 12:31:53 INFO Worker: Executor app-20210814123152-0000/0 finished with state EXITED message Command exited with code 134 exitStatus 134

From reading issues like #328 and others it sounds like such an approach is not intended or at least not supported at the time. Furthermore I understand the difficulty of dynamically scraping potentially random ports.

So I am primarily asking for advice of how to approach this or if there is some easy setup I am missing?

@dupetr
Copy link

dupetr commented Oct 26, 2022

Hello @fstab @tomwilkie,
I have a working solution to this issue - We had nearly the same case. We have Apache Spark executors running on EMR clusters and we have like 5 executors running per machine. In the current agent's config, you define 1 port and that doesn't work when you have N number of JVMs trying to create a webserver on the same port.

We have an agent, to which you can give range of ports as an config and it will try to start webservers on a port in this range.

Current agent is very simple:

  1. read config
  2. start server

I had to "mess it up" a bit, in order to accommodate some retries and backoffs. Before I can submit a PR, I need to go trough an internal review process.

My question is, what form and shape should the contribution be? Right now the code changes are in jmx_prometheus_javaagent_common. I have preserved the original code flow when you define 1 port. Would prefer to have the changes https://github.com/prometheus/jmx_exporter/blob/main/jmx_prometheus_javaagent_common/src/main/java/io/prometheus/jmx/JavaAgent.java or perhaps create a different class or even put it to a different module?

@scMarkus
Copy link
Author

scMarkus commented Oct 27, 2022

Indeed having multiple webservers is a similar issue like i describIed. But I think you still need to know the multiple ports in advance?
My workaround so far has been to write a tremendously simple custom prometheus exporter which is part of our spark worker docker container. It lists and scrapes every jvm process inside the container. Therefore this is not an agent anymore but small standalone application (works with every jvm process)

@dupetr
Copy link

dupetr commented Oct 30, 2022

But I think you still need to know the multiple ports in advance?

Yes you need to know the range you're targeting, have some headroom there and set the rest of the system to that range.

In the spark-submit extra java options the command went from
-javaagent:./jmx_prometheus_javaagent-0.17.2.jar=1000:config.yaml
to
-javaagent:./jmx_prometheus_javaagent-0.17.2.jar=1000-1010:config.yaml

There will be some wasted ports in this solution.

@dhoard
Copy link
Collaborator

dhoard commented May 16, 2023

Not sure I agree with the approach. The general approach is to use the Pushgateway and push metrics for these types of use cases (https://prometheus.io/docs/instrumenting/pushing/)

If you are submitting a Spark job where the Java agent HTTP port is dynamic, how are you monitoring it?

With that said... I feel a better implementation would be...

  1. Take the contiguous port range and create a Stack of all available ports
  2. Shuffle the Stack (Collections.shuffle(stack))
  3. Process the Stack, popping the next port value or System.exit if the Stack is empty (port range exhausted)
  4. Create an InetSocketAddress with the port
  5. Use the HTTPServer.Builder withInetSocketAddress method to create the HTTPServer
  6. If there is an Exception, loop back to 3, else continue.

... but again, I don't agree with the approach.

@dupetr
Copy link

dupetr commented May 19, 2023

I recall thinking about something a bit more smarter than just going one-by-one from portStart. I like the idea with stack, definitely looks more professional, although I'm not sure if it would improve startup time significantly. When starting up a Spark job with 700-1000 JVMs, it's never instantaneous and it takes a bit of a time for everything to start (speaking from a point when the job is marked as running by YARN). That said, there was no noticeable delay with the JMX Agent.

If you are submitting a Spark job where the Java agent HTTP port is dynamic, how are you monitoring it?

The way it is then scraped will depend on infrastructure layout. In our case, there is no direct communication between EC2 instance and Prometheus. We need to explicitly allow communication for it. It isn't and cannot be truly dynamic as you need to know beforehand which ports to allow in Security Groups / ACLs.
In Prometheus you have a target per port from the range. Let's say we have target_exporter_1000 until target_exporter_1010. Each of those targets will contain a fraction of the metrics, depending on how it played out on the EMR. You cannot really influence what happens on the EMR, you just need to make sure that you won't have more JVMs per node than your defined port range.

@scMarkus
Copy link
Author

I mostly forgot about this issue. Thanks for bringing it up again.
At this point in time my workaround has been to colocate a jvm_exporter with every spark worker process which connects to all visible jvm processes and scrapes jmx metrics from those exporting them in Prometheus format as a single server. This means Prometheus -> jvm_exporter -> jvm_xx (where jvm_xx can be multiple executor processes). This has turned out to be relatively light weight and so far has been functioning without any issues (except the somewhat tricky configuration). Furthermore I would assume this being not bound to spark but could function with any multi jvm setup.

If some one else is having the same issue and willing to implement this workaround I might be able to share the code.

@retpolanne
Copy link

I was wondering if this could be achieved with service location protocol?

4.1. Service: URLs

   Service URL syntax and semantics are defined in  [13].  Any network
   service may be encoded in a Service URL.

   This section provides an introduction to Service URLs and an example
   showing a simple application of them, representing standard network
   services.

   A Service URL may be of the form:

      "service:"<srvtype>"://"<addrspec>

   The Service Type of this service: URL is defined to be the string up
   to (but not including) the final `:'  before <addrspec>, the address
   specification.

   <addrspec> is a hostname (which should be used if possible) or dotted
   decimal notation for a hostname, followed by an optional `:'  and
   port number.

I'm trying to get a good grasp of java's jmx, rmi and slp, but I believe jmx servers could advertise their service URL with port to an RMI server and then from that server jmx-exporter could find those services and their ports and use them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants