Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage information too high for large esbuild produced bundle #502

Open
jakebailey opened this issue Nov 22, 2023 · 6 comments
Open

Coverage information too high for large esbuild produced bundle #502

jakebailey opened this issue Nov 22, 2023 · 6 comments

Comments

@jakebailey
Copy link
Contributor

jakebailey commented Nov 22, 2023

  • Version: v20.9.0
  • Platform: Linux JABAILE-DESK02 5.15.90.1-microsoft-standard-WSL2 Detailed coverage #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

I can only apologize in advance for this one...

I work on TypeScript, and I've been wanting to set up coverage information for the compiler. c8 does a great job; significantly faster than nyc (of course). I use c8 in my own projects with success.

However, capturing coverage for the compiler produces results that are too high to believe. checker.ts is a 50k line file, and yet its coverage is:

checker.ts | 99.99 | 95.38 | 98.62 | 99.99 | 50577-50578

There's no way that we actually have tested every line of the checker. And indeed, adding a function to a known-dead path (like one ending in Debug.assertNever) indicates that the line is covered when it definitely is not, and the overall count doesn't change.

image

I don't know if I have a good repro for this besides npx c8 npm test -- --no-lint on our repo. Obviously that's huge but I don't really know where to start.

At best, I can observe that npx c8 npm test -- --no-lint --no-bundle works better, i.e. not using esbuild and instead running the tsc output produces a more believable coverage count:

checker.ts  |    97.5 |    95.87 |   98.62 |    97.5 | ...-48684,48686-48687,48761-48762,48 ...
@jakebailey
Copy link
Contributor Author

One thing that is interesting is that comparing the bundled output HTML page with the unbundled output HTML page, the paths are different. Specifically, the bundled output starts right with compiler/checker.ts, whereas the unbundled output has src/compiler/checker.ts.

@jakebailey
Copy link
Contributor Author

Yeah, the unbundled output is way better:

image

@ericmorand
Copy link

@jakebailey did you happen to solve the issue? I have the same problem with bundled test suite where, as soon as a portion of a source file is covered, the whole file is marked as covered.

@jakebailey
Copy link
Contributor Author

No, besides using #521.

@ericmorand
Copy link

Thanks a lot. I tried it and it mostly solve the issue. I'll comment there, with a minimal reproducer repository to, hopefully, help tackle the remainder issues.

@gerardo-lima-moonfare
Copy link

Me too I was having 100% coverage when running tests using tsx, which uses esbuild under the hood:

tsx --enable-source-maps --test-reporter=spec --test 'src/**/*.spec.ts'

After some tests, I changed to @swc-node register:

node --import @swc-node/register/esm-register --enable-source-maps --test-reporter=spec --test 'src/**/*.spec.ts'

My preliminary conclusion is: the entry points for the tests were the test files, themselves, and this made any non-tested code to be marked as unreachable by eslint (and removed from the report). This would lead to 100% coverage, by definition.

This was not an exhaustive research, but it makes sense to me and the change produced accurate test coverage reports. I would be interested to understand if other people got different results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants