Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FATAL ERROR]: JavaScript heap out of memory #365

Closed
ild0tt0re opened this issue Nov 26, 2023 · 15 comments
Closed

[FATAL ERROR]: JavaScript heap out of memory #365

ild0tt0re opened this issue Nov 26, 2023 · 15 comments
Labels
bug Something isn't working

Comments

@ild0tt0re
Copy link

ild0tt0re commented Nov 26, 2023

Description

I tried Knip in different company projects (Nx integrated-monorepo based). In the small one it works correctly and we were able to leverage the potential of the tool 🚀 but where it would be most useful like in large monorepos (with 34_875 files or more) It fails for JavaScript heap out of memory

Command

Tried with both, knip latest 2.43.0

$ pnpm exec knip

and the canary version 3.0.0-canary.4

$ bunx knip@canary

the result is the same:

Output

Analyzing source files...

<--- Last few GCs --->

[25564:0x130008000]   326662 ms: Mark-Compact 4038.8 (4139.6) -> 4024.9 (4141.4) MB, 1334.46 / 0.00 ms  (average mu = 0.156, current mu = 0.025) allocation failure; scavenge might not succeed
[25564:0x130008000]   327980 ms: Mark-Compact 4040.9 (4141.4) -> 4027.4 (4143.9) MB, 1262.71 / 0.00 ms  (average mu = 0.104, current mu = 0.042) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Software Overview

  • knip: 2.43.0 or 3.0.0-canary.4
  • nx: 17.0.3
  • nodejs: 20.9.0
  • OS: Sonoma 14.1.1

Hardware Overview:

  • Model Name: MacBook Pro
  • Model Identifier: Mac14,7
  • Chip: Apple M2
  • Total Number of Cores: 8 (4 performance and 4 efficiency)
  • Memory: 16 GB
@ild0tt0re ild0tt0re added the bug Something isn't working label Nov 26, 2023
@ild0tt0re ild0tt0re changed the title [] [FATAL ERROR]: JavaScript heap out of memory Nov 26, 2023
@webpro
Copy link
Collaborator

webpro commented Nov 26, 2023

Thanks for putting this together. Unfortunately there's not much I can do from this information alone. I need a reproduction.

Things you could try:

  • Give Node.js more memory with e.g. NODE_OPTIONS=--max-old-space-size=5120 npx knip - tweak the number for your machine.
  • Use --workspace [dir] to focus on specific a workspace.

Something I want to try:

One thing Knip does is some sort of "workspace merging", based on tsconfig.json settings: different basePath and paths may result in more "workspace workers" internally. This results in less programs for the TypeScript backend, speeding up the process overall. I want to see whether this optimization could optionally be turned off to prevent errors you're seeing, at the expense of speed. But again, this requires rep(r)os that fail to even start looking into it and try it out.

@webpro
Copy link
Collaborator

webpro commented Nov 28, 2023

I've added an undocumented --isolate-workspaces flag, but I think it only uses more memory, not less. Any chance you could give it a shot? Use Knip v3 and knip --isolate-workspaces.

@ild0tt0re
Copy link
Author

Things you could try:

  • Give Node.js more memory with e.g. NODE_OPTIONS=--max-old-space-size=5120 npx knip - tweak the number for your machine.

I tried with NODE_OPTIONS="--max-old-space-size=8192" npx knip but It didn't work ❌


npx knip --workspace libs/my-package/root_dir/

It works ✅
But is this way reliable? doesn't the tool miss the context of the whole workspace and thus return false positives?


Something I want to try:

One thing Knip does is some sort of "workspace merging", based on tsconfig.json settings: different basePath and paths may result in more "workspace workers" internally. This results in less programs for the TypeScript backend, speeding up the process overall. I want to see whether this optimization could optionally be turned off to prevent errors you're seeing, at the expense of speed. But again, this requires rep(r)os that fail to even start looking into it and try it out.

Obviously I can't share the company projects but I am at your disposal for testing 😉


I've added an undocumented --isolate-workspaces flag, but I think it only uses more memory, not less. Any chance you could give it a shot? Use Knip v3 and knip --isolate-workspaces.

Same error ❌

bunx [email protected] --isolate-workspaces
Analyzing source files...

<--- Last few GCs --->

[20239:0x158008000]   138753 ms: Mark-Compact 4049.5 (4138.0) -> 4044.7 (4141.5) MB, 1437.21 / 0.00 ms  (average mu = 0.508, current mu = 0.037) allocation failure; scavenge might not succeed
[20239:0x158008000]   141567 ms: Mark-Compact 4052.8 (4141.5) -> 4048.0 (4144.7) MB, 2790.33 / 0.00 ms  (average mu = 0.238, current mu = 0.008) allocation failure; scavenge might not succeed

<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

@webpro
Copy link
Collaborator

webpro commented Nov 29, 2023

Bummer... but thanks a lot for trying!

But is this way reliable? doesn't the tool miss the context of the whole workspace and thus return false positives?

It should be reliable, you can read about it here: https://knip.dev/features/monorepos-and-workspaces/#lint-a-single-workspace

@onskulit
Copy link

onskulit commented Dec 4, 2023

I tried with NODE_OPTIONS="--max-old-space-size=8192" npx knip but It didn't work ❌

I faced such problem too. Somehow process of knip does not receive new value of NODE_OPTIONS. You can check it in your script with node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'.

But this works:

export NODE_OPTIONS='--max-old-space-size=8192' && npx knip

@ild0tt0re ild0tt0re reopened this Dec 4, 2023
@ild0tt0re
Copy link
Author

I tried with NODE_OPTIONS="--max-old-space-size=8192" npx knip but It didn't work ❌

I faced such problem too. Somehow process of knip does not receive new value of NODE_OPTIONS. You can check it in your script with node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'.

But this works:

export NODE_OPTIONS='--max-old-space-size=8192' && npx knip

Same error

<--- Last few GCs --->

[75481:0x158008000]   166946 ms: Mark-Compact (reduce) 8084.1 (8239.5) -> 8083.5 (8240.0) MB, 6313.54 / 0.00 ms  (average mu = 0.324, current mu = 0.001) allocation failure; scavenge might not succeed
[75481:0x158008000]   173210 ms: Mark-Compact (reduce) 8084.6 (8240.0) -> 8083.7 (8240.2) MB, 6259.96 / 0.00 ms  (average mu = 0.182, current mu = 0.001) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

@webpro
Copy link
Collaborator

webpro commented Dec 5, 2023

Thanks for chiming in, @onskulit!

I was wondering if it's possible there's a potential circular dependency causing infinite recursion or otherwise recursive calls exhausting memory. I have done some work and try to prevent that from happening, but could be buggy or incomplete.

E.g. #359 was opened because of a "RangeError: Maximum call stack size exceeded"

Would be great if anyone could look into that, to rule out that the high memory usage is caused by infinite recursion. I mean, if raising the memory for the process does not help, there might be recursive calls going on (probably in the context of multiple workspaces) that leads to the fatal error. I'm not sure how to try and reproduce it myself.

@webpro
Copy link
Collaborator

webpro commented Dec 6, 2023

@ild0tt0re Any chance you could try this for me?

  1. Since findReferences is an expensive function and is called more often if there are many class and enum members (near the end of the process), I would like to know if that's where things start to go wrong or not. Does Knip still explode when you run this:
knip -n --performance --exclude classMembers,enumMembers
  1. Perhaps it's interesting to see how far in the process Knip gets by running in debug mode:
knip --debug

Would there be parts of the output you could share (redacted)? I'm especially interested to see how many "programs" Knip is using (the debug lines with [Pn/n] as shown at https://knip.dev/guides/performance/#workspace-sharing), and how far it gets into the process in general. Whatever you can share might help :)

@ild0tt0re
Copy link
Author

@ild0tt0re Any chance you could try this for me?

  1. Since findReferences is an expensive function and is called more often if there are many class and enum members (near the end of the process), I would like to know if that's where things start to go wrong or not. Does Knip still explode when you run this:
knip -n --performance --exclude classMembers,enumMembers

Same error:

<--- Last few GCs --->
[88202:0x140008000]   168421 ms: Mark-Compact 8054.7 (8226.9) -> 8046.0 (8234.4) MB, 4988.92 / 0.00 ms  (average mu = 0.608, current mu = 0.071) allocation failure; scavenge might not succeed
[88202:0x140008000]   173779 ms: Mark-Compact 8061.9 (8234.4) -> 8052.2 (8240.2) MB, 5297.00 / 0.00 ms  (average mu = 0.397, current mu = 0.011) allocation failure; scavenge might not succeed

<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
  1. Perhaps it's interesting to see how far in the process Knip gets by running in debug mode:
knip --debug

Would there be parts of the output you could share (redacted)? I'm especially interested to see how many "programs" Knip is using (the debug lines with [Pn/n] as shown at https://knip.dev/guides/performance/#workspace-sharing), and how far it gets into the process in general. Whatever you can share might help :)

d/index.tsx',
                                                                  ]
[*] Analyzing used resolved files [P60/1] (7)
[
              ]
[*] Analyzing used resolved files [P61/1] (16)
[
                                ]
[*] Analyzing used resolved files [P62/1] (54)
[
                                                                                                            ]
[*] Analyzing used resolved files [P63/1] (63)
[
                                                                                                                              ]
[*] Analyzing used resolved files [P64/1] (16)
[
                                ]
[*] Analyzing used resolved files [P65/1] (43)
[
                                                                                      ]
[*] Analyzing used resolved files [P66/1] (92)
[
                                                                                                                                                                                        ]
[*] Analyzing used resolved files [P67/1] (116)
[
                                                                                                                                                                                                                                        ]
[*] Analyzing used resolved files [P68/1] (12)
[
                        ]
[*] Analyzing used resolved files [P69/1] (20)
[
                                        ]
[*] Analyzing used resolved files [P70/1] (135)
[
                                                                                                                                                                                                                                                                              ]
[*] Analyzing used resolved files [P71/1] (10)
[
                    ]
[*] Analyzing used resolved files [P72/1] (23)
[
                                              ]
[*] Analyzing used resolved files [P73/1] (39)
[
                                                                              ]
[*] Analyzing used resolved files [P74/1] (9)
[
                  ]
[*] Analyzing used resolved files [P75/1] (55)
[
                                                                                                              ]
[*] Analyzing used resolved files [P76/1] (6)
[
            ]
[*] Analyzing used resolved files [P77/1] (4)
[
        ]
[*] Analyzing used resolved files [P78/1] (40)
[
                                                                                ]
[*] Analyzing used resolved files [P79/1] (196)
[
                                                                                                                                                                                                                                                                                                                                                                                                        ]
[*] Analyzing used resolved files [P80/1] (158)
[
                                                                                                                                                                                                                                                                                                                            ]
[*] Analyzing used resolved files [P81/1] (531)
[
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      ]
[*] Analyzing used resolved files [P82/1] (30)
[
                                                            ]
[*] Analyzing used resolved files [P83/1] (11)
[
                      ]
[*] Analyzing used resolved files [P84/1] (23)
[
                                              ]
[*] Analyzing used resolved files [P85/1] (18)
[
                                    ]
[*] Analyzing used resolved files [P86/1] (42)
[
                                                                                    ]
[*] Analyzing used resolved files [P87/1] (17)
[
                                  ]
[*] Analyzing used resolved files [P88/1] (25)
[
                                                  ]
[*] Analyzing used resolved files [P89/1] (72)
[
                                                                                                                                                ]
[*] Analyzing used resolved files [P90/1] (29)
[
                                                          ]
[*] Analyzing used resolved files [P91/1] (171)
[
                                                                                                                                                                                                                                                                                                                                                      ]
[*] Analyzing used resolved files [P92/1] (127)
[
                                                                                                                                                                                                                                                              ]
[*] Analyzing used resolved files [P92/2] (1)
[
  ]
[*] Analyzing used resolved files [P93/1] (134)
[
                                                                                                                                                                                                                                                                            ]
[*] Analyzing used resolved files [P94/1] (10)
[
                    ]
[*] Analyzing used resolved files [P95/1] (46)
[
                                                                                            ]
[*] Analyzing used resolved files [P96/1] (3)
[

]
[*] Analyzing used resolved files [P97/1] (221)
[
                                                                                                                                                                                                                                                                                                                                                                                                                                                          ]
[*] Analyzing used resolved files [P97/2] (12)
[
                        ]
[*] Analyzing used resolved files [P98/1] (450)
[
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ]
[*] Analyzing used resolved files [P99/1] (124)
[
                                                                                                                                                                                                                                                        ]
[*] Analyzing used resolved files [P100/1] (289)
[
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  ]

<--- Last few GCs --->

[98476:0x148008000]   106549 ms: Mark-Compact 4034.4 (4134.8) -> 4025.4 (4140.1) MB, 2098.25 / 0.00 ms  (average mu = 0.470, current mu = 0.089) allocation failure; scavenge might not succeed
[98476:0x148008000]   109464 ms: Mark-Compact 4041.6 (4140.3) -> 4030.9 (4144.8) MB, 2880.12 / 0.00 ms  (average mu = 0.251, current mu = 0.012) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100a82114 node::Abort() [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
 2: 0x100a822fc node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
 3: 0x100c09048 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
 4: 0x100ddd82c v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
 5: 0x100ddc308 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
 6: 0x100dd2b20 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
 7: 0x100dd3380 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
 8: 0x100db8384 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
 9: 0x10119fd94 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
10: 0x1014fcc44 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
11: 0x106f8ef8c
12: 0x1078b60dc
13: 0x1076da57c
14: 0x10772a45c
15: 0x1076c90f0
16: 0x107f515c0
17: 0x10789a038
18: 0x107369f6c
19: 0x10761ff4c
20: 0x106d78e88
21: 0x10768ca24
22: 0x107861f4c
23: 0x1080944d0
24: 0x1080982ac
25: 0x106c286d0
26: 0x1072ce610
27: 0x106c1ee44
28: 0x106c2974c
29: 0x1072ce610
30: 0x106c1ee44
31: 0x106c2974c
32: 0x1072ce610
33: 0x106c1ee44
34: 0x106c2974c
35: 0x1072ce610
36: 0x106c1ee44
37: 0x106c2974c
38: 0x1072ce610
39: 0x106c1ee44
40: 0x106c2974c
41: 0x1072ce610
42: 0x106c1ee44
43: 0x106c2974c
44: 0x1072ce610
45: 0x106c1ee44
46: 0x106c2974c
47: 0x1072ce610
48: 0x106c1ee44
49: 0x106c2974c
50: 0x1072ce610
51: 0x106c1ee44
52: 0x106c2974c
53: 0x1072ce610
54: 0x106c1ee44
55: 0x106c2974c
56: 0x1072ce610
57: 0x106c1ee44
58: 0x106c2974c
59: 0x1072ce610
60: 0x106c1ee44
61: 0x106c2974c
62: 0x1072ce610
63: 0x106c1ee44
64: 0x106c2974c
65: 0x1072ce610
66: 0x106c1ee44
67: 0x106c2974c
68: 0x1072ce610
69: 0x106c1ee44
70: 0x106c2974c
71: 0x1072ce610
72: 0x106c1ee44
73: 0x106c2974c
74: 0x1072ce610
75: 0x106c1ee44
76: 0x106c2974c
77: 0x1072ce610
78: 0x106c1ee44
79: 0x106c2974c
80: 0x1072ce610
81: 0x106c1ee44
82: 0x106c2974c
83: 0x1072ce610
84: 0x106c1ee44
85: 0x106c2974c
86: 0x1072ce610
87: 0x106c1ee44
88: 0x106c2974c
89: 0x1072ce610
90: 0x107298be4
91: 0x106d76b24
92: 0x107996b84
93: 0x106eb0850
94: 0x10791bcd0
95: 0x1078e761c
96: 0x106f0f0fc
97: 0x106f18aa8
98: 0x106f1a1d8
99: 0x106f0fbc4
100: 0x106f372f8
101: 0x106e6fe68
102: 0x1014ab210 Builtins_AsyncFunctionAwaitResolveClosure [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
103: 0x101558fb8 Builtins_PromiseFulfillReactionJob [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
104: 0x10149ab94 Builtins_RunMicrotasks [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
105: 0x1014723f4 Builtins_JSRunMicrotasksEntry [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
106: 0x100d4ac94 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
107: 0x100d4b180 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
108: 0x100d4b35c v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
109: 0x100d72418 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
110: 0x100d72bb4 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
111: 0x101475ef0 Builtins_CallApiCallback [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
112: 0x107a31a44
113: 0x10147250c Builtins_JSEntryTrampoline [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
114: 0x1014721f4 Builtins_JSEntry [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
115: 0x100d4acbc v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
116: 0x100d4a108 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
117: 0x100c249d4 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
118: 0x1009b4d3c node::InternalCallbackScope::Close() [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
119: 0x1009b501c node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
120: 0x1009cb48c node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
121: 0x100a877c8 node::fs::FSReqCallback::Reject(v8::Local<v8::Value>) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
122: 0x100a880a4 node::fs::FSReqAfterScope::Reject(uv_fs_s*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
123: 0x100a884c4 node::fs::AfterStat(uv_fs_s*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
124: 0x100a7d210 node::MakeLibuvRequestCallback<uv_fs_s, void (*)(uv_fs_s*)>::Wrapper(uv_fs_s*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
125: 0x101450d4c uv__work_done [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
126: 0x10145479c uv__async_io [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
127: 0x101466874 uv__io_poll [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
128: 0x101454d60 uv_run [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
129: 0x1009b5754 node::SpinEventLoopInternal(node::Environment*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
130: 0x100ac5138 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
131: 0x100ac4ed4 node::NodeMainInstance::Run() [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
132: 0x100a4d60c node::Start(int, char**) [/Users/myuser/.volta/tools/image/node/20.9.0/bin/node]
133: 0x181ded0e0 start [/usr/lib/dyld]

@webpro
Copy link
Collaborator

webpro commented Dec 8, 2023

Thanks a lot, how many workspaces do you have in your monorepo? Is it actually 100 or more?

@ild0tt0re
Copy link
Author

ild0tt0re commented Dec 8, 2023

Thanks a lot, how many workspaces do you have in your monorepo? Is it actually 100 or more?

There are 106 projects:

  • 103 libs
  • 3 apps

@ild0tt0re
Copy link
Author

ild0tt0re commented Dec 8, 2023

@webpro if can be helpful below you can find the Nodejs memory profiling:

Sampling profile:

-> Heap-20231208T173058.heapprofile.zip

How to load the file:

Chrome DevTools -> Memory tab -> right click on the left sidebar -> load -> select the unzipped file

Preview

Screenshot 2023-12-08 at 17 45 57

@webpro
Copy link
Collaborator

webpro commented Dec 15, 2023

Thanks for all the help, @ild0tt0re! It was really helpful.

This issue was one of the reasons I was motivated even more to work on lowering memory usage and improving performance. I wrote about it (at https://knip.dev/blog/slim-down-to-speed-up/) and you can install knip@canary to try it out if you like.

There's a lot more plans in this area, so even if it doesn't work out today it might be soon. E.g. caching and recovery should be helpful features as well. Also, the size of projects like yours might give false positives, but at least, hopefully, there will be some useful output.

(By the way, I couldn't find anything related to Knip in the memory heap profile, there's no matches when I grep for e.g. "knip" or even "typescript". It all seems related to npm cli. I might be misunderstanding the purpose, though.)

@ild0tt0re
Copy link
Author

Thanks for all the help, @ild0tt0re! It was really helpful.

This issue was one of the reasons I was motivated even more to work on lowering memory usage and improving performance. I wrote about it (at https://knip.dev/blog/slim-down-to-speed-up/) and you can install knip@canary to try it out if you like.

Hey @webpro you did it! 🏆 👏 👏 👏 👏 👏
I was able to run the tool only with npx knip@canary without NODE_OPTIONS or other hackery.

There's a lot more plans in this area, so even if it doesn't work out today it might be soon. E.g. caching and recovery should be helpful features as well. Also, the size of projects like yours might give false positives, but at least, hopefully, there will be some useful output.

I cant't wait! A reliable and high-performance tool like this is exactly the missing piece in the JS/TS community

(By the way, I couldn't find anything related to Knip in the memory heap profile, there's no matches when I grep for e.g. "knip" or even "typescript". It all seems related to npm cli. I might be misunderstanding the purpose, though.)

Thanks for the check!

@webpro
Copy link
Collaborator

webpro commented Dec 15, 2023

I'm so glad to hear this! Thanks for all the feedback, let's rock on 🚀

@webpro webpro closed this as completed Dec 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants