Sidecar fails after reaching 4GB RSS Memory

[Imod7]

Description
A team using Sidecar has reported issues where the Sidecar processes gradually consumes up to 4Gb of RSS memory and then becomes incapable of handling any requests, requiring the process to restart. It's running in its own container alongside a Polkadot node. Despite allocating more resources to the container, it consistently fails once it hits the 4GB mark.

Linked to #1361

Possible Issues
Memory Leaks ?

Possible Good Resources

• Node.js Memory Limits - What You Should Know
• Node.js memory management in container environments

[filvecchiato]

While the container has more resources to allow Sidecar to run, if the process is not assigned a larger heap it would crash when the limit is reached. The start command could be modified to allow more heap memory (to match a larger container).

Some of the work done between PJS and Sidecar should allow it to run more efficiently so decreasing the likelihood to this specific case happening, however, more work is required to reduce the memory consumption. The PJS cache system will still be a big contributor to this issue.

[gituser]

After updating from v19.0.2 to v19.3.1 sidecar failed in 24 hours with error:

Nov 30 03:20:04 polkadot sidecar[577113]: <--- Last few GCs --->
Nov 30 03:20:04 polkadot sidecar[577113]: [577113:0x62e3480] 115832777 ms: Mark-sweep 3784.2 (4142.6) -> 3767.8 (4142.6) MB, 107.0 / 0.0 ms  (average mu = 0.337, current mu = 0.328) allocation failure; scavenge might not succeed
Nov 30 03:20:04 polkadot sidecar[577113]: [577113:0x62e3480] 115832931 ms: Mark-sweep 3784.0 (4142.9) -> 3767.7 (4142.9) MB, 107.4 / 0.0 ms  (average mu = 0.318, current mu = 0.298) task; scavenge might not succeed
Nov 30 03:20:04 polkadot sidecar[577113]: <--- JS stacktrace --->
Nov 30 03:20:04 polkadot sidecar[577113]: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Nov 30 03:20:04 polkadot sidecar[577113]:  1: 0xb83f50 node::Abort() [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]:  2: 0xa94834  [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]:  3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]:  4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]:  5: 0xf42265  [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]:  6: 0xf43168 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]:  7: 0xf53673  [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]:  8: 0xf544e8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]:  9: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]: 10: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]: 11: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]: 12: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
Nov 30 03:20:04 polkadot sidecar[577113]: 13: 0x17035b9  [/usr/bin/node]

The container running sidecar has a plenty of memory, previously v19.0.2 has been running without issues for 6 months at least.

Here is a workaround, run sidecar with this option max-old-space-size:

/usr/bin/node --max-old-space-size=8192 /home/polkadot/sidecar-current/src/main.js

[Imod7]

Thank you @gituser so much for your feedback! 💯
Yes, the --max-old-space-size is one of the workarounds we also suggested in the issue #1555. Sharing which exact memory settings/amount works for you is super useful. We will keep it in mind for other users and might also update the README to give more visibility. Thanks again!

[gituser]

Thank you @gituser so much for your feedback! 💯 Yes, the --max-old-space-size is one of the workarounds we also suggested in the issue #1555. Sharing which exact memory settings/amount works for you is super useful. We will keep it in mind for other users and might also update the README to give more visibility. Thanks again!

@Imod7

Unfortunately 8192MB is not enough for sidecar, I'll continue increasing it, but it's not normal and this bug should be fixed.
Also, interesting note that after OOM sidecar starts working and getting this specific block works again.
Most likely there is a memory leak somewhere in the sidecar, as I noted earlier v19.0.2 worked just fine without any issues.

[gituser]

Any update on this? @Imod7 @filvecchiato

v19.0.2 works just fine and v19.3.1 fails after running about 20 hours. It's clearly a regression in the code somewhere.

[TarikGul]

Any update on this? @Imod7 @filvecchiato

v19.0.2 works just fine and v19.3.1 fails after running about 20 hours. It's clearly a regression in the code somewhere.

Just to clarify, you haven't tried any other versions between 19.0.2 and 19.3.1 right? I haven't dived into this much, but I am happy to profile some of the releases in between 19.0.2, and 19.3.1 when I am back on Monday. I would be glad to give a helping hand on this and see it get resolved.

I would be curious to see if this is related to polkadot-js in any way as well.

[gituser]

Any update on this? @Imod7 @filvecchiato
v19.0.2 works just fine and v19.3.1 fails after running about 20 hours. It's clearly a regression in the code somewhere.

Just to clarify, you haven't tried any other versions between 19.0.2 and 19.3.1 right? I haven't dived into this much, but I am happy to profile some of the releases in between 19.0.2, and 19.3.1 when I am back on Monday. I would be glad to give a helping hand on this and see it get resolved.

@TarikGul No, I didn't try versions in between. But I think v19.3.0 is affected as well.

I would be curious to see if this is related to polkadot-js in any way as well.

Will be waiting for your results! Thanks.

[Imod7]

In this HackMD I reviewed all the changes found between v19.0.2 and v19.3.1 of Sidecar and categorized them based on their criticality and potential performance impact. I will continue to investigate and will soon add memory usage measurements for selected versions.

[gituser]

Hey @Imod7 ! I see there is a new version released v20.0.0 but in the changelog there is no indication that this bug has been fixed yet.

Can you clarify a progress please on this issue.

Thanks.

[Imod7]

Hello @gituser !
Yes, there is quite some progress profiling Sidecar, focusing on two areas: 1) execution time analysis in function-level, 2) heap allocation per Sidecar version. I plan to make all documents with the results public and will update this thread with all the relevant links by end of day. However, the exact root cause of this issue is still TBD. Are you still experiencing the same issue on your end?

EDIT : I documented everything in this HackMD, made all the related docs public and will try to move this issue up in the priority list. Until now some PRs were merged related to performance but work still needs to be done to complete the profiling and implement more changes.

[gituser]

This issue is still actual on the latest release of polkadot (stable2503) and sidecar (sidecar-v20.2.0).

Any update on this? It's been quite a while, on the old versions of sidecar in the production it works just fine.

EDIT: noticed there is a new v20.2.1 sidecar version, just updated to it, but pretty sure it will reproduce on this version too.

[gituser]

This issue is still actual on the latest polkadot and sidecar releases:
sidecar-v20.2.2
polkadot-stable2503

[klefevre]

Hi, we're encountering the same issue. Still actual on v20.3.2.