FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed

[robertsLando]

This error afetr 1 minute running a benchmark against aedes with mongodb backend (it happens both with and without clusters). Thoughts @mcollina ?

Seems the bottle-neck is mongodb insert method.

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1376259]
Security context: 0x235d15b008a1 <JSObject>
    1: startSession [0x3f5e4fa6e611] [/media/daniel/DATA/Git-Projects/aedes-tests/node_modules/mongodb/lib/core/sdam/topology.js:~466] [pc=0x2301c4388df4](this=0x0aaea0c76b41 <EventEmitter map = 0x20d775fe4b51>,0x15a404a905d1 <Object map = 0x3d3208614891>,0x304da9a004a9 <undefined>)
    2: emitPacket [0x8ab9be098e1] [/media/daniel/DATA/Git-Projects/aedes-tests/no...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200429.113415.30558.0.001.json
Node.js report completed
 1: 0x9dab70 node::Abort() [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
 2: 0x9dbd26 node::OnFatalError(char const*, char const*) [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
 3: 0xb3b18e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
 4: 0xb3b509 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
 5: 0xce68a5  [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
 6: 0xce6f36 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
 7: 0xcf2dca v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
 8: 0xcf3cd5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
 9: 0xcf66e8 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
10: 0xcbd017 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType) [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
11: 0xff307b v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]
12: 0x1376259  [/home/daniel/.nvm/versions/node/v12.14.1/bin/node]

<--- Last few GCs --->

[30557:0x47309b0]   233807 ms: Mark-sweep 2038.3 (2051.1) -> 2037.9 (2051.3) MB, 1800.6 / 0.0 ms  (average mu = 0.084, current mu = 0.028) allocation failure scavenge might not succeed
[30557:0x47309b0]   235702 ms: Mark-sweep 2038.5 (2051.3) -> 2038.0 (2051.6) MB, 1863.6 / 0.0 ms  (average mu = 0.050, current mu = 0.016) allocation failure scavenge might not succeed

If I use aedes-persistence-mongodb with mqemitter-redis everything works fine and performances are even better then using only redis persitence.

It even works with mqemitter-mongodb paired with aedes-persistence-redis but with worse performances

I think mqemitter-mongodb and aedes-persistence mongodb needs some thoughts about a way to optimize inserts in order to fix this

[mcollina]

I think there is a memory leak in mqemitter-mongodb under high load, and it lives here: https://github.com/mcollina/mqemitter-mongodb/blob/798d191ce36e6ac8dac39a3b8d1713aba632bba2/mqemitter-mongodb.js#L170-L215.
Last time I checked, I was not able to track it.

[robertsLando]

I think that error could cause some message lost but not what I’m sepaking about, am I wrong?

…

--- Daniel - Software Engineer Support me at: Github sponsors

On 29 Apr 2020, at 12:26, Matteo Collina ***@***.***> wrote:  I think there is a memory leak in mqemitter-mongodb under high load, and it lives here: https://github.com/mcollina/mqemitter-mongodb/blob/798d191ce36e6ac8dac39a3b8d1713aba632bba2/mqemitter-mongodb.js#L170-L215. Last time I checked, I was not able to track it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[mcollina]

I think whenever a message is lost it leaks memory.

[robertsLando]

Are you speaking about this line? https://github.com/mcollina/mqemitter-mongodb/blob/798d191ce36e6ac8dac39a3b8d1713aba632bba2/mqemitter-mongodb.js#L213

[robertsLando]

Could we instead of using mongodb object id use a simple incremental id?

[mcollina]

I don't think it would change the problem. I don't think that callback is called when a message is missed.