Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Leak on AWS ECS #927

Closed
antlionguard opened this issue Oct 10, 2023 · 8 comments
Closed

Memory Leak on AWS ECS #927

antlionguard opened this issue Oct 10, 2023 · 8 comments

Comments

@antlionguard
Copy link

antlionguard commented Oct 10, 2023

Environment



Reproduction

I cannot reporduce because that's very huge project. I just wanna trace where memory leak happens.

Describe the bug

Here's the memory utilization on AWS ECS Monitoring section. The parts where the decrease occurs are when the container is restarted due to out of memory.

image

Additional context

And how can i trace where memory leaks come from? Any tips for tracing?

Logs

error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Aborted (core dumped)
17: 0x15d9e19 [/usr/local/bin/node]
16: 0x11f9038 v8::internal::Runtime_RegExpExec(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
15: 0x11d3e67 v8::internal::RegExpImpl::IrregexpExec(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int, v8::internal::Handle<v8::internal::RegExpMatchInfo>, v8::internal::RegExp::ExecQuirks) [/usr/local/bin/node]
14: 0x11d3cee v8::internal::RegExpImpl::IrregexpPrepare(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
13: 0x11d30f0 v8::internal::RegExpImpl::CompileIrregexp(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, bool) [/usr/local/bin/node]
12: 0x11d2947 v8::internal::RegExpImpl::Compile(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::RegExpCompileData*, v8::base::Flags<v8::internal::JSRegExp::Flag, int>, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::String>, bool, unsigned int&) [/usr/local/bin/node]
11: 0x11b45b9 v8::internal::RegExpCompiler::Assemble(v8::internal::Isolate*, v8::internal::RegExpMacroAssembler*, v8::internal::RegExpNode*, int, v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
10: 0x14c8d8e v8::internal::RegExpMacroAssemblerX64::GetCode(v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
8: 0xe7a215 v8::internal::Factory::CodeBuilder::BuildInternal(bool) [/usr/local/bin/node]
9: 0xe7ac2e v8::internal::Factory::CodeBuilder::Build() [/usr/local/bin/node]
7: 0xe79bb6 v8::internal::Factory::CodeBuilder::AllocateCode(bool) [/usr/local/bin/node]
6: 0xeabe58 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
4: 0xce1d77 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
5: 0xe993e5 [/usr/local/bin/node]
3: 0xce19d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
1: 0xb090e0 node::Abort() [/usr/local/bin/node]
2: 0xa1b70e [/usr/local/bin/node]

<--- Last few GCs --->
[28:0x54111c0] 146309520 ms: Mark-sweep (reduce) 741.0 (771.3) -> 741.0 (767.1) MB, 1014.7 / 0.0 ms (average mu = 0.724, current mu = 0.025) last resort GC in old space requested
[28:0x54111c0] 146310521 ms: Mark-sweep (reduce) 741.0 (767.1) -> 741.0 (766.6) MB, 1001.5 / 0.0 ms (average mu = 0.565, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Copy link
Member

Would you see if you can reproduce with just nitro and, if so, open an issue on https://github.com/unjs/nitro instead? 🙏

@antlionguard
Copy link
Author

Would you see if you can reproduce with just nitro and, if so, open an issue on https://github.com/unjs/nitro instead? 🙏

I'm not using nitro in this project. Nitro is set false in bridge config. But i'm gonna try with nitro and vite. I hope i can make it. There's lots of error occurs when i set nitro and vite set true.

@danielroe danielroe removed the upstream label Oct 10, 2023 — with Volta.net
Copy link
Member

danielroe commented Oct 10, 2023

Oh, if you're not using nitro then the memory leak will probably still reproduce if you enable it.

What makes you think this is an issue with bridge? Does it reproduce outside AWS ECS?

(I would recommend trying with meta: false btw.)

@antlionguard
Copy link
Author

Oh, if you're not using nitro then the memory leak will probably still reproduce if you enable it.

What makes you think this is an issue with bridge? Does it reproduce outside AWS ECS?

(I would recommend trying with meta: false btw.)

Project runs into docker container in ecs. But i cannot reproduce in my local docker enviroment. This makes things quite interesting. When i watching memory usage in node inspect everything is normal. Garbage collector works fine in my local. I'm gonna try with meta: false and I will inform you what will happen next. Thanks for quick replies <3

@antlionguard
Copy link
Author

antlionguard commented Oct 10, 2023

Maybe the reason why I cannot reproduce it in Local env is that this process takes too long. Because when I look at the chart, the process of resetting the container takes approximately 15-20 hours.

image

@antlionguard
Copy link
Author

meta: false seems work. I will continue to monitor for 1-2 more days and share the results.

image

@antlionguard
Copy link
Author

probably meta: false solved the problem. What can I provide that might help you resolve the issue caused by meta:true? @danielroe

@antlionguard
Copy link
Author

There's no memory leak witn latest nuxt-bridge and node 20.10.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants