AxeBuilder.analyze allocates/leaks ~3 MiB per call

[johannespfrang]

Describe the bug

Since introducing axe tests throughout our entire SPA, we've been battling 2 GiB OOM errors on our CI runners. I can trace approximately 1.2 MiB per call to leaking this large axe script:

in the TestStats output array:

Various other strings in that structure make up most of the remainder of the 3 MiB per-call allocation (I think it's mostly violations which take up the remaining space, and one window.partialResults ??= ''; ... script).

To Reproduce
We can reproduce the issue as described in #1044. I've also pushed a simplified reproducer to https://github.com/johannespfrang/axe-wdio-memory-leak.

Expected behavior

No memory leaks. The TestStats should, if at all, only contain test stats, and not duplicate the same long scripts (as strings) hundreds/thousands of times.

Screenshots
Provided above

Environment (please include versions for all products, browsers, OS, etc used ):
@axe-core/webdriverio 4.9.1
@wdio/spec-reporter 8.39.0
WebdriverIO 8.39.1
Chrome for Testing 126.0.6478.182
Ubuntu 24.04

[johannespfrang]

AFAICT, the script lands in the TestStats structure this way:

1. @axe-core/webdriverio uses execute / executeAsync to execute the (rather large) script(s).
2. WebdriverIO internally executes a protocol command, which emits, among other data, the "body" (script) here.
3. That object is then emitted to the configured reporters here.
4. And persisted in the TestData there.

Not sure if there's any way to stop that from happening...

[johannespfrang]

@christian-bromann Hey Christian, sorry to bother you, but would you consider this behavior a bug in WebdriverIO itself, or here in @axe-core/webdriverio? I don't see a a way for Axe to do anything else here, and the "leak" actually happens in the reporter's TestStats. See the reproduction repository here for details, as well as this issue.

Maybe it makes sense to only save the beginning of the script body in the TestStats, or maybe the reporter could deduplicate identical scripts somehow. If Axe would only inject the script(s) once that would also solve this instance, but from what I can tell they can't do that (and it would only help with SPAs where the script can stay around in the global scope).

[christian-bromann]

This certainly seems like a bug in WebdriverIO, feel free to raise it there instead.

[johnp]

So, webdriverio/webdriverio#13219 would deal with ~1.2 MiB/call due to the main Axe script. The remainder has more complicated causes:

1. The recursive chunkResults can create a lot of large, unique scripts, which cannot be deduplicated.
   • Experimentally, (preferentially) chunking by the partialResults array elements (passed as arguments to execute instead of interpolated into the script) looks very promising.
2. @axe/webdriverio bypasses the WebDriver result (de)serialization and instead uses JSON.stringify here. Since those large (1-3 MiB, depending on what Axe finds) result JSON strings are unique, deduplicating them is not possible either. Note that while the comment blames WebdriverIO, it's actually the WebDriver specification which prescribes this behavior, and the actual (chrome|edge|gecko)driver which does this.
   • Experimentally, simply removing the custom (de)serialization shows a significant improvement in memory usage, especially when there's a lot to report.
3. @axe/webdriverio also does not use the WebDriver argument serialization and instead JSON.stringifys potentially large(?) context structures (and other objects) into (thereby unique) scripts.
   • Experimentally, using arguments instead of script interpolation didn't seem to improve things there, at least on top of fix(@wdio/reporter): transform scripts in TestStats (#13209) webdriverio/webdriverio#13219. (with the notable exception of point 1.)

[christian-bromann]

would deal with ~1.2 MiB/call due to the main Axe script.

There has been discussion at the W3C to support "pinning scripts" which in Bidi is now available through the preload script command. I would recommend to maybe take a look at this option where you maybe inject an Axe scripts once in the beginning of the session where you attach the function to the window environment and then just have a small execute payload that essentially triggers the execution.