Memory Leak related to io.netty.util.concurrent.FastThreadLocal in Vertx code #5078
Comments
|
If someone would need code to debug this situation I will share my as example Add this dependencies to ByteBuddy library Instrument related code with something like this Prepare some debug information and log it or get by some debug API like I did |
|
And example how OldGen heap looks like with ZGC on Java 21 with this problem |
|
thanks @bolbat |
|
FYI, I think you could the the leak detection of a sync profiler (adding --live to the cmd line options) to detect this or just allocation profiling. Thanks for the analysis, although by reading this I don't see the problem in the fast thread local, but instead in the high number of combiner executors created: any idea why @vietj ? Said that, fast thread locals could be "smarter" in resizing the array, if it knows that a combiner is no longer reachable, and could do a better job. |
|
Confirming that problem was fixed in 4.5.2-SNAPSHOT Heap OldGen graph for last 2 days from one node: |
Related to eclipse-vertx/vert.x#5078 FastThreadLocal can cause a memory-leak if not used as a class constant. Signed-off-by: Thomas Segismont <[email protected]>
Related to eclipse-vertx/vert.x#5078 FastThreadLocal can cause a memory-leak if not used as a class constant. Signed-off-by: Thomas Segismont <[email protected]>
Version
Vertx: bug since 4.4.x, available on 4.5.1
Netty: version doesn't matter, tested with latest 4.1.106.Final
Java: tested on 17 and 21, but could be reproduced on any supported version
Context
FastThreadLocal feature designed to be faster than basic ThreadLocal and internally use InternalThreadLocalMap which internally have:
Size of indexedVariables field depends on a static nextIndex field
If someone create a lot of FastThreadLocal instance - this global static index will be increased by one for each instance
As result for all InternalThreadLocalMap instances each invocation of setIndexedVariable method will resize indexedVariables array if index is bigger than array size (resizing indexedVariables arrays in all instance because of global static index)
Consequently all FastThreadLocalThread instances (including VertxThread) and FastThreadLocal fields will have continuously increasing indexedVariables arrays in their InternalThreadLocalMap instances till JVM will be restarted
Our case
On our production load one instance serving tens thousands requests per second and doing much more outgoing requests
Each instance use 64 physical cores and use 128 threads for Netty event-pool
For around 8h uptime each of this 128 threads have own indexedVariables array instance with length 8_000_000+ and use 60MB+ memory
99.999% values in each array instance is empty and filled with InternalThreadLocalMap.UNSET object
Usually each thread / array store only 3-15 values from features that use FastThreadLocal in a right way
Heap dump details
What caused this?
After some investigation I've found that there some instances of FastThreadLocal stored as not static fields and could be created a lot of times if parent object also created a lot of times
How to find exactly which code does this?
I've implemented debug feature (with ByteBuddy constructor interceptor) for FastThreadLocal and collected StackTraces stats with information about who creating this instances
It looks like this
Now we see that problem is related to HttpClient component and especially to io.vertx.core.net.impl.pool.CombinerExecutor
It's generating a lot of instances and as consequence created a lot of FastThreadLocal instances
Cause of this is the next commit: f15cf1c
In scope of this PR: #4750
And this ticket: #4749
Additional information
I've checked who potentially can also cause this problem if someone change code without understanding FastThreadLocal implementation details
Already problem or high risk:
Can be a problem:
Good examples how to use FastThreadLocal feature in a safe way (just use as static final fields)
The text was updated successfully, but these errors were encountered: