[QUESTION] java.lang.OutofMemoryError:GC overhead limit exceeded

[conorzhao]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

Occasional GC overhead exception after running for a period of time

Affects Version(s)

1.7.1

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[github-actions[bot]]

Hello @conorzhao,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

[pan3793]

A question without clear and enough information is not likely to get active responses.

[conorzhao]

java.lang.OutofMemoryError:GC overhead limit exceeded 
    at java.lang.Class.getDeclaredMethods0 (Native Method) 
    at java.lang.Class.privateGetDeclaredMethods (Class.java:2701)
    at java.lang.Class.getDeclaredMethod(Class.java:2128) 
    at java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1542)
    at java.io.ObjectStreamClass.access$2400(ObjectStreamClass.java:79)
    at java.io.ObjectStreamClass$2.run(objectStreamClass.java:521)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:482)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamclass.<init>(ObjectStreamClass.java:482)
    at java.io.ObjectStreamClass.lookup(objectStreamClass.java:379)
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:669)
    at java.io.ObjectInputStream.readordinaryobject (ObjectInputStream.java:2037)
    at java.io.ObjectInputStream.readobjecto(ObjectInputStream.java:1568)
    at java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java:2282)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
    at java.io.ObjectInputStream.readordinaryobject (objectInputStream.java:2064)
    at java.io.ObjectInputStream.readobjecto(objectInputStream.java:1568)
    at java.io.ObjectInputStream.readobject (ObjectInputStream.java:428) 
    at org.apache.spark.serializer.JavaDeserializationStream.readobject (JavaSerializer.scala:87)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:123)
    at org.apache.spark.rpc.netty.NettyRpcEnv.Sanonfun$deserialize$2(NettyRpcEnv.scala:299)
    at org.apache.spark.rpc.netty.NettyRpcEnv$Lambda$1155/1602652077.apply(Unknown Source)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) 
    at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize (NettyRpcEnv.scala:352) 
    at org.apache.spark.rpc.netty.NettyRpcEnv.Sanonfun$deserialize$1(NettyRpcEnv.scala:298)
    at org.apache.spark.rpc.netty.NettyRpcEnv$$Lambda$1154/558968000.apply(Unknown Source)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) 
    at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:298)
    at org.apache.spark.rpc.netty.RequestMessage$.apply(NettyRpcEnv.scala:646)

After the user's kyuubi engine is started for a period of time, submitting task jobs will throw a GC ahead exception. After restarting the engine, it will resume.
There are no tasks running in the engine

[conorzhao]

A question without clear and enough information is not likely to get active responses.

Thank you, I will provide more detailed information

[conorzhao]

It seems that the kyuubi engine is unable to properly GC and release memory

[pan3793]

After the user's kyuubi engine is started for a period of time, submitting task jobs will throw a GC ahead exception

There is no causal relationship between the two. A Kyuubi Spark engine won't get OOM without heavy workloads even after lives ten years. The memory usage in Spark is quite complex, if you have no idea, please analyze JVM memory using conventional methods first.

PS: paste text instead of pictures as much as possible, the latter is not searchable thus may not found be by future explorers.

[conorzhao]

The phenomenon is that Kyuubi is not running empty and then ooming, but has been running for a period of time and submitting many Spark jobs. However, the Spark jobs have already been completed, and submitting Spark tasks to Kyuubi in the future will throw this exception. In theory, it has nothing to do with Spark tasks anymore, but Kyuubi's engine application can no longer receive tasks

[pan3793]

Converted to discussion since it's not a bug but a question

[conorzhao]

After the user's kyuubi engine is started for a period of time, submitting task jobs will throw a GC ahead exception

There is no causal relationship between the two. A Kyuubi Spark engine won't get OOM without heavy workloads even after lives ten years. The memory usage in Spark is quite complex, if you have no idea, please analyze JVM memory using conventional methods first.

PS: paste text instead of pictures as much as possible, the latter is not searchable thus may not found be by future explorers.

[conorzhao]

Occasionally, I will dump the memory again next time I encounter it

[pan3793]

If you insist it's related to GC, consider reducing the periodic GC interval

• spark.cleaner.periodicGC.interval

  default value: 30min

  Controls how often to trigger a garbage collection. This context cleaner triggers cleanups only when weak references are garbage collected. In long-running applications with large driver JVMs, where there is little memory pressure on the driver, this may happen very occasionally or not at all. Not cleaning at all may lead to executors running out of disk space after a while.

https://spark.apache.org/docs/latest/configuration.html

[conorzhao]

thx