perf: monomorphized objects change in ts 5.5 significantly slows down third-party tooling

[walkerdb]

🔎 Search Terms

typescript 5.5 performance, slowdown, typescript-eslint

🕗 Version & Regression Information

• This changed between versions 5.4 and 5.5
• This changed in commit or PR Monomorphize allocators for tsserver/public API, just like core compiler #58045 (verified by running our type-aware lints while running git bisect on a local typescript repo, it slows down specifically on this commit)

⏯ Playground Link

No response

💻 Code

I've put together a repro in a forked version of the sentry monorepo here: walkerdb/sentry#1. It's using the sentry repo only because it's a large public repo that uses typescript-eslint.

🙁 Actual behavior

When running typescript-eslint's type-aware lint rules in a large monorepo with typescript 5.5, we observe lint times between 1.3x to 3x worse than when running the exact same rules and the same config under typescript 5.4. The slowdown seems to be entirely coming from the monomorphized objects change in the ts 5.5 release.

🙂 Expected behavior

There should be no performance impact when moving from typescript 5.4 to 5.5.

Additional information about the issue

We've submitted a bug report to typescript-eslint but we're also reporting here since the slowdown can be pinpointed to a specific typescript change.

• typescript-eslint issue: Bug: using with typescript v5.5 slows down linting by 2.3x typescript-eslint/typescript-eslint#9474
• the PR that introduces the slowdown: Monomorphize allocators for tsserver/public API, just like core compiler #58045

Also attaching some before/after update perf traces via 0x that show the extra time is coming largely from typescript internals. The highlighted boxes show the specific part of the process that has ballooned in runtime. I'm happy to share the actual traces with maintainers if they'd be useful in debugging.

ts 5.4	ts 5.5

[DanielRosenwasser]

I didn't get that far on this today, but I believe this is from more time spent in the GC. If you try upping your --max-old-space-size to something way higher, let me know if that alleviates things.

As an aside, SO much time in your build is being spent on shims running within eslint-plugin-react.

While findVariableByName really should be optimized to not create an intermediate array with variablesInScope, is it possible that you are running on an undesirable input file?

[jakebailey]

This is the opposite of expected! Which walk function is that? I'm not familiar with that "walk" function. Traces would be helpful, but walkerdb/sentry#1 is probably enough to give us a lead.

You may also consider running things via https://www.npmjs.com/package/pprof-it (what Daniel just used above) as it in my experience does a better job attributing time and memory allocations to specific lines of code.

[jakebailey]

Hm, my trace doesn't seem to show that particular code as to blame:

@walkerdb What tool did you use to find your result?

[jakebailey]

I timed this, and in 5.5, it's 100s, but 5.4 is 91s; not 2-3x slower from what I can tell...

[jakebailey]

A second run of 5.4 was 100s too. Not sure what's going on here. What version of Node are you using?

[jakebailey]

I was using Node 22; switching down to Node 18 shows a worse regression:

// 5.4.5
total time:  160.75s
user time:   212.16s
system time: 4.96s
CPU percent: 135%
max memory:  3946 MB

// 5.5.3
total time:  217.70s
user time:   292.33s
system time: 6.36s
CPU percent: 137%
max memory:  3995 MB

Not 2-3x but very much farther apart than Node 22.

Unfortunately I'm now going to be gone until next week, so this will just have to eat me up inside until then...

[walkerdb]

Hi @jakebailey! Sorry, I should have clarified a few things:

1. the traces above were generated using 0x, but on our private monorepo, which has a 2.5x slowdown when running our typed lints under ts 5.5 instead of 5.4. I used sentry as a public repro but its slowdown was only ~1.3x on my machine.
2. in our private monorepo we run lint with essentially unlimited max-old-space and also with higher max-semi-space since we run into gc issues with young gen / scavenge as well at our size.
3. we are indeed running node 18! Likely will update to 22 by end of year but that's a major undertaking for us.

to @DanielRosenwasser I'm not a part of sentry, I was just using that repo as an example repro since ours is private.

[walkerdb]

the types in Notion's private monorepo can be quite complex, and I suspect are close to a pathological case for some typescript perf behavior, which may help explain the 2.5x slowdown we were observing. I'll see if I can hack running the same checks under node 22 to see if the timings change there much.

[jakebailey]

Ran Node 18, and the results are very different from #59101 (comment):

That is an incredible amount of GC time, along with the badness Daniel noted above. Given the difference between Node 18 and Node 22 on this being in that shim code, I'm not 100% certain that this repro is representative of the repo you're testing internally, unfortunately...

Would you be able to run pprof-it on your own codebase? Screenshots of flame graphs and such should not expose any details, but you can also run it with PPROF_SANITIZE=on to sanitize the output for sharing (as that will remove any paths from the profile).

[walkerdb]

@jakebailey I'll give pprof-it a try, ty!

[walkerdb]

here's my pprof result for typescript 5.5.2 on node 18.18. I'll also upload one for typescript 5.4 in a sec
pprof-time-typescript-55.pb.gz
pprof-heap-typescript-55.pb.gz

[walkerdb]

pprof results for the exact same run but using typescript 5.4.5 on node 18.18 . This is significantly faster, and matches our expected run time from prior to the 5.5 update
pprof-time-typescript-54.pb.gz
pprof-heap-typescript-54.pb.gz

[walkerdb]

last run, with typescript 5.5.2 and node 22.3.0. Running under a newer node version doesn't seem to actually be much faster in this case?
pprof-time-typescript-55-node-22.pb.gz
pprof-heap-typescript-55-node-22.pb.gz

[jakebailey]

In all of those profiles, nearly half the time is spent just getting/setting the node children via the WeakMap. Yikes. I'm not sure why this isn't observable outside of your internal code unless WeakMaps get noticably slower when they get massive.

May have to tack that prop back on again, but the prop not being on all nodes directly was one of the things we had to do to not blow up perf too much.

[jakebailey]

At least I assume it's the weak map itself; haven't quite figured out how to remap the paths in the profile to a real version. (I'm also on a plane, that doesn't help)

[walkerdb]

thanks so much @jakebailey and @DanielRosenwasser for digging in! Happy to help however I can, feel free to shoot me a message at [email protected] if there's any other info you could use / commands to run / code to share / etc that I can help with.

In the meantime though we can wait through the holiday week, hope you have a great vacation!

[walkerdb]

can confirm that the getChildren call in the trace maps to this code in the ts source

TypeScript/src/services/services.ts

Line 445 in ddf43cd

public getChildren(sourceFile?: SourceFileLike): readonly Node[] {

here also are some pprof diff screenshots between the 5.4.5 run and the 5.5.2 run, generated with go tool pprof -http=:8080 --diff_base=pprof-time-typescript-54.pb.gz pprof-time-typescript-55.pb.gz

[walkerdb]

Interesting also that the slower ts 5.5 run uses significantly less heap, 23gb for 5.4 vs just 14gb on 5.5. Most of the items in the screenshots are typescript-eslint functions that themselves call ts public methods. (screenshots from a similar diff view comparing 5.4 heap usage to 5.5 heap usage, with 5.4 as the baseline)

[DanielRosenwasser]

@walkerdb any chance you can iterate and reduce this repro to see if there's a root cause beyond just scale?

For example, if you cut out half of the files in your program - even if there are build errors, do you see the same surprising amount of time in set/getNodeChilren? If so, could you keep trying that until it stops? I'd be curious to see if there's a specific set of files that causes it, or if it's just fundamentally something about the move to WeakMaps.