Research and implement a testing strategy for types provided through @carbon/react

[tay1orjones]

Background

Once #12513 is fully complete, it would be beneficial if we had a way to be more confident in future changes to the component types. Specifically to ensure that types provided through @carbon/react follow the established Semantic Versioning guidelines for the package.

Important

Types will continue to be provided on an as-is basis and remain exempt from semver for the forseeable future as we establish a more robust framework around maintaining types. Completing the work in this issue does not mean that we're able to bound types to semver.

Click to expand details

carbon/docs/guides/versioning.md

Lines 75 to 86 in b8701c9

	#### A change is made to component typings/definitions
	semver bump: **minor**
	Typescript definitions for component APIs are provided on an as-is basis and are
	not bound to semver. This includes type additions, changes, or removals.
	Breaking changes may happen across patch or minor versions. The component API
	may shift and types could become outdated.
	If a typing is out of date or incorrect, please let us know by opening an issue
	or even better, propose a fix via PR! We appreciate the community's involvement
	in this effort to keep component types up to date and as correct as possible!

Potential approaches

Snapshot the type interfaces

For component proptypes we have a very helpful setup where the proptypes for all components are collected and snapshotted into a single file, PublicAPI-test.js.snap. As changes to proptypes are made, the snapshot requires being updated, which shows up in the PR diff. This makes it really clear what's changing and makes it much easier to spot potential issues. We could attempt to mirror this setup in some way to provide more explicit visibility on every PR when a potentially semver-incompatible change is made to component types. The prominent solution for this would be to use https://api-extractor.com/

Implicitly test types in storybook

If we converted our story files to TypeScript we'd be "testing" the component types. It wouldn't cover all use cases but it would be a quick way to get closer to how an end user/consumer actually uses the types. It should only be slightly more overhead to write story files in typescript, and could encourage greater TypeScript familiarity amongst our community contributors.

Directly test the types

This approach has probably the most overhead. Options to research include:

• tsd - Seems like a popular option. The programmatic api in particular seems like a way we could automate this testing instead of having to write individual tests for each component.
• https://github.com/JoshuaKGoldberg/eslint-plugin-expect-type
• https://github.com/mmkal/expect-type
• https://github.com/TypeStrong/ts-expect
• Use Vitest, it supports testing types. This could have lots of additional concerns as it would be adding a whole new framework to our testing strategy.
• https://github.com/tstyche/tstyche

Additional concerns

• We should try to avoid having to manually write and maintain tests, instead favoring automation where possible
• Would be awesome to have as much coverage as possible, including testing generics, even though they may only be rarely used in the codebase.
• ...

Related resources

• https://www.totaltypescript.com/how-to-test-your-types
• https://github.com/microsoft/DefinitelyTyped-tools/tree/main/packages/dtslint#just-looking-for-a-type-testing-tool
• https://github.com/JoshuaKGoldberg/eslint-plugin-expect-type?tab=readme-ov-file#references
• An interesting comparison of projects: https://github.com/mmkal/expect-type?tab=readme-ov-file#similar-projects

Tasks

Preview Give feedback

1. Testing types: Research and explore available options
2. Testing types: Provide a written summary of the research, indicating a preferred approach
3. Testing types: Socialize options with other Carbon library maintainers
4. Testing types: Implement the chosen solution, including adding it to the CI workflows

[mrazauskas]

@tay1orjones It is really nice to see TSTyche in the list. I am its author. Long story short, I was writing and contributing type tests to Jest repo for some time. At the moment there are more than 1000 assertions. Working at this scale was a pain with the available tooling. That is why I wrote TSTyche.

For instance, at large scale there was a need to have describe() and test() helpers with .skip and .only. That’s a must for deducing an issue or building new feature. Also we wanted to test on different versions of TypeScript easily. TSTyche can do all this and much more. All Jest’s type test now are ran by TSTyche.

Differently from all other type testing tools, TSTyche is wrapping around TypeScript’s language service. It allows per file queries instead of working with the whole TS program. And more. For example, another feature I am thinking about is an API which makes possible testing in memory generate files, i.e. passing arbitrary text to the compiler and asking for diagnostics (and/or processing type test assertions).

If this sounds like the automation solution you are looking for, I am more than interested to collaborate. The implementation on TSTyche side is rather trivial. The fs host object must be passed to the language service factory. Obviously, that can be a virtual fs which is generating test files on demand.

To be honest, I never had a chance to work with component type tests. And that sounds even more interesting ;D

[mrazauskas]

Earlier today TSTyche 2.1 shipped with new .toAcceptProps() matcher that is designed to test well.. prop types of JSX components! Thanks for inspiration.

[tay1orjones]

@mrazauskas Thanks for popping in here! The features you listed are definitely intriguing and something we'll consider when we take up this work. toAcceptProps in particular could be really valuable.

[github-actions]

Thank you for submitting a feature request. Your proposal is open and will soon be triaged by the Carbon team.

If your proposal is accepted and the Carbon team has bandwidth they will take on the issue, or else request you or other volunteers from the community to work on this issue.