Test against older versions of React

[tay1orjones]

This issue will track the status of improving our test suite to include testing against multiple versions of React to ensure compatibility.

The problem

Right now the matrix of React versions officially supported in @carbon/react is React 16, 17, and 18.

carbon/packages/react/package.json

Lines 40 to 42 in 5c26f2f

	"peerDependencies": {
	"react": "^16.8.6 || ^17.0.1 || ^18.2.0",
	"react-dom": "^16.8.6 || ^17.0.1 || ^18.2.0",

Currently, the CI checks only test against the version of React we use in development, v18.

carbon/packages/react/package.json

Lines 111 to 112 in 5c26f2f

	"react": "^18.2.0",
	"react-dom": "^18.2.0",

Proposed solution

Improve our matrix of testing to include some type of testing using previous/future versions of react.

Open questions

• Do we have enough bugs with React 16 and 17 to warrant improving this coverage?
• What's the best way to update our CI to run against multiple versions?
  • An interesting example here: https://github.com/SAP/ui5-webcomponents-react/pull/787/files#diff-bdd00ef7ca7fea0d5b8e26a84ec28cf441b7fd06a7f343231e8027acdfd0cb9d
• Do our existing test suites pass in React 16 and 17?
  • Does @testing-library/react work in React <18? If not, is it feasible to downgrade this dependency along with React deps without having to rewrite or modify tests or their config?
  • Do we instead need to avoid our existing tests and write new smoke tests for running specifically in React 16 and 17?
• How can we avoid duplicate snapshotting? We don't want to run vrt multiple times
• Should @avt tests be ran against multiple versions of react?
• How much will this increase CI time? Do we need to Improve yarn build time with distributed caching #14244 first?

Steps to ensure compatibility

Preview Give feedback

1. Update CI to run the base @testing-library/react-based unit tests against React 16 and 17
2. Determine if e2e @avt tests should be ran against multiple versions of react, and implement if so

[tay1orjones]

Concerns with this approach that were discussed today:

• Some testing library versions (and other deps) only support certain react versions, and the syntax is different
• Older dep versions often only support older/unsupported versions of node, or drop support for certain node versions
• So tests (and the testing environment/deps) would need to be rewritten for react 15,16,17,18
• We can't afford to rewrite tests in this way and configure the differing specific (older) supported node environments.
• If we instead focus on new integration tests, we could put together some playwright tests to cover older versions of react. These would have to be just smoke tests though and not really extensive or cover all scenarios or component configurations.
• If we compare the scale of doing all this work with the surface area of actual bugs we see due to differing react versions, it doesn't seem like this would pay off:
  • Usage of Carbon with React 16 is very low, ~4.9%
  • Usage of Carbon with React 17 is low, ~26.6%
  • Usage of Carbon with React 18 is the highest at ~68.35%
  • Usage of Carbon with React 19 is ~0.15%

Overall the consensus is that this work is not necessary at this time. Instead we will focus our efforts on fixing bugs that emerge in older versions of react as they are reported by users.

That said, #13610 is still valid and should remain open and something we should prioritize.