[Feature] Feature Flags & Future Defaults

[jdanil]

• I'd be willing to implement this feature (contributing guide)
• This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

As a yarn maintainer, (I imagine) it can be tedious managing long pre-release cycles for major versions.

• Pull requests that introduce breaking changes may be blocked until sufficient changes are built up to warrant a major release, leading to longer development cycles and a need to maintain several feature branches (i.e. rebase and resolve conflicts).
• Preparing a major release would involve reviewing TODO comments in the code and GitHub issues and pull requests to find mentions of changes slated for a major release.
• Once a major RC is cut, releasing a new minor/patch version would require cherry picking commits (e.g. updating the TypeScript patch).
• Major versions can have long pre-release cycles until all proposed breaking changes are complete.

As a yarn contributor, it can be difficult to validate features before releasing them.

• Pull requests would need to be largely feature complete before merging. This can lead to the need for very long lived feature branches while large/complex features are iterated on.
• While yarn allows you to set your version to a particular branch via yarn set version from sources this takes you off the main release track, and the number of people that would use this version as a daily driver to trial or beta test features is limited.
• Post-release, if a user uncovers an issue that requires a breaking change it would need to wait until the next major release.

As a yarn consumer, long release cycles may lead to more difficult migrations.

• Once a major RC is cut, I am unable to take advantage of minor/patch changes (unless they are cherry picked and released) without upgrading to the RC.
• It can be difficult to upgrade to a major RC without release notes (i.e. going through the changelog and commits, which may not document migration steps).
• A large number of changes are built up in major releases, potentially making upgrades more time consuming.
• I can only use one patched version of yarn at a time (e.g. if there are two branches with fixes/features I want to take advantage of, I have to choose between them).

Describe the solution you'd like

I would like to propose using feature flags for some breaking changes.

Yarn already provides a lot of customisation via the yarnrc file, but I don't believe it has been commonly used for feature flags.

The yarnrc file could be extended to include something like a new "flags" object to house breaking and experimental changes.

I really like webpack's pattern for this where experiments are included in the main releases, and can be opted-into by users with the understanding that these experiments may contain breaking changes and their behaviour may change in future minor/patch releases.

They also provide a useful flag to enable future defaults which may enable several other experiments and also include warnings for anything that may be problematic in future major releases.

I believe feature flags would help address some of the above issues with the following benefits.

For yarn maintainers:

• Future breaking changes could be merged behind feature flags, avoiding the need for long lived feature branches or TODO comments.
• A smaller number of breaking changes would need to be "blocked" (ones that may not be able to be easily hidden behind a feature flag).
• Minor/patch releases can continue on the master branch while major version updates are underway.
• Preparing major releases could be streamlined by simply searching for experiment names/keys to find where they are used, and removing the conditionals and legacy code paths.

For yarn contributors:

• Features will be more readily accessible for users to trial and provide feedback.
• Changes can marinate behind a feature flag, allowing contributors to make breaking changes before an official release.
• Large changes can be made iteratively across several PRs, without the need to maintain long lived feature branches.

For yarn consumers:

• Major versions will be less disruptive (by virtue of having fewer changes built up).
• Future defaults can be opted-into to smooth migration paths.
• "experimental" changes can be trialed without being taken off the main release track.
• Several "experimental" changes can be trialed at the same time.

Looking back at the breaking changes included in v4 so far, I would guess that at least the following changes could have been released behind feature flags.

• Removes the compatibility layer for Clipanion 2 #4262
• Make the local cache opt-in (rather than opt-out) #4252
• Prefer using exclusively Corepack when possible #4254
• Implements --refresh-lockfile #4299
• chore(pnp)!: remove fallbacks to react-scripts and gatsby #3004
• feat(foreach)!: automatically enable --verbose in tty environments #4595
• feat!: set default nmMode to hardlinks-local #4504
• fix(core)!: rename caFilePath to httpsCaFilePath #4503
• fix(audit)!: remove fallback to publish registries #4639
• refactor(sdks)!: drop fallback on createRequireFromPath #4697
• refactor(init)!: disable zero-installs by default #4698
• refactor!: stop removing old .pnp.js files when migrating #4774

I would expect the development effort needed to support feature flags in yarnrc would be minimal. The bigger change would be the process change to use feature flags for breaking changes where appropriate.

Describe the drawbacks of your solution

Introducing feature flags will require the addition of conditionals to the code and more branches, increasing the complexity of the code. These would need to remain until the next major version.

The testing strategy needs to be considered. It would be unreasonable for all combinations of feature flags to be tested in all test cases. Should tests default to enabling all feature flags to ensure future compatibility and avoid the need for widespread test updates when preparing a major version? Or should tests default to disabling all feature flags to ensure compatibility with the current major version?

Would need to be cautious in code reviews to not let feature flags make merging of half-baked changes permissible.

Describe alternatives you've considered

Alternative would be the status quo.