Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Potential Memory Leak #11007

Open
sdelamo opened this issue Jul 23, 2024 · 3 comments
Open

Potential Memory Leak #11007

sdelamo opened this issue Jul 23, 2024 · 3 comments

Comments

@sdelamo
Copy link
Contributor

sdelamo commented Jul 23, 2024

Issue description

A user upgrading from 3.9.2 to 4 pointed to a potential memory leak

Our application (running on micronaut-platform 4.4.5) is crashing with an out of memory error.

After some analysis of the heap dump, we've found that the issue is due to DirectByteBuffers not being released properly.

We've then enabled this flag to track netty byte buffers:

-Dio.netty.leakDetectionLevel=paranoid

The logs seems to report that the error happens while writing a json response in the micronaut http server:

2024-07-17 13:15:27.048 Jul-17 11:15:27.046 [default-nioEventLoopGroup-5-9] 86.144.114.222 ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.

Recent access records:

#1:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:617)
	io.netty.buffer.ByteBufOutputStream.write(ByteBufOutputStream.java:80)
	com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2210)
	com.fasterxml.jackson.core.json.UTF8JsonGenerator.close(UTF8JsonGenerator.java:1234)
	com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4804)
	com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3999)
	io.micronaut.jackson.databind.JacksonDatabindMapper.writeValue(JacksonDatabindMapper.java:207)
	io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:161)
	io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:377)
	io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:355)
	io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:249)
	io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:94)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
	io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
	reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2812)
	reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
	reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
	io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
	io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
	io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
	io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
	io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
	io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
	io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
	io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
	io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
	io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
	io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
	java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	java.base/java.lang.Thread.run(Thread.java:840)
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
	io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
	io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:158)
	io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:377)
	io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:355)
	io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:249)
	io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:94)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
	io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
	reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2812)
	reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
	reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
	io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
	io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
	io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
	io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
	io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
	io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
	io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)

We've tried to update to the latest available micronaut version 4.5.0, to check if the issue was solved already, and run the application in a separate environment, but the leak is still detected in the same place.

2024-07-15 15:58:23.630 Jul-15 13:58:23.629 [io-executor-thread-6] 82.61.68.148 ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.

Recent access records:

#1:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:617)
	io.netty.buffer.ByteBufOutputStream.write(ByteBufOutputStream.java:80)
	com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2210)
	com.fasterxml.jackson.core.json.UTF8JsonGenerator.close(UTF8JsonGenerator.java:1234)
	com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4804)
	com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3999)
	io.micronaut.jackson.databind.JacksonDatabindMapper.writeValue(JacksonDatabindMapper.java:207)
	io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:163)
	io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:380)
	io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:358)
	io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:248)
	io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:101)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
	io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
	reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2834)
	reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
	reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
	io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
	io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
	io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
	io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
	io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
	io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
	io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
	io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
	io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
	io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
	io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
	java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	java.base/java.lang.Thread.run(Thread.java:840)
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
	io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
	io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:160)
	io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:380)
	io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:358)
	io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:248)
	io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:101)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
	io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
	reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
	reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2834)
	reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
	reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
	io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
	io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
	io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
	io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
	io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
	io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
	io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
	io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
	io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)

The same request doesn't always produce the log, but we see a steady increase in the size of the memory used by the application.

We are using jackson-databind for serialization and we cannot move to the new micronaut serialization approach just yet.

Inspecting the micronaut codebase, we've found that the buffer allocated in

micronaut-core/http-netty/src/main/java/io/micronaut/http/netty/body/NettyJsonHandler.java

Line 160 in df3a78f

ByteBuf buffer = nettyContext.alloc().buffer();
is not released immediately but this is delegated to the DefaultFullHttpResponse#release() method but we where not able to track it further.
@graemerocher
Copy link
Contributor

We will need an example application

@sdelamo
Copy link
Contributor Author

sdelamo commented Jul 23, 2024 •
edited
Loading

@yawkat when creating the ByteBufOutputStream, we are not using the constructor which releases on close . I assume that it is intentional because we pass the ByteBuff to DefaultFullHttpResponse and we expect someone to invoke DefaultFullHttpResponse::release. Who is doing that?

@yawkat
Copy link
Member

yawkat commented Jul 24, 2024

Yes it's intentional. PipeliningServerHandler does the release. This is not sufficient information to determine the cause of the leak.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@graemerocher @sdelamo @yawkat