A new dependency hell? peerDependencies and virtual packages

[jeffrson]

Well, I'm not sure, where's the right place to ask or discuss this problem. I already tried Discord, but unfortunately it does not seem to be the best place for such questions. While I got help to solve my immediate problem, I'm not sure this is a real solution.

In fact, our team wonders, if we can rely on yarn berry at all, because its behaviour related to virtual packages bothers us a lot - it looks like a systematic flaw (or shortcoming) in design. Although the reasoning sounds understandable, its practical implications appear to be huge.

We are using "moleculer", a microservices framework, accompanied by "moleculer-decorators". While moleculer has a lot of (optional) peer dependencies, molecular-decorators, which itself depends on moleculer, does not depend on any of those.
So we have a common service library (depending on moleculer) and a couple of services (also depending on moleculer, and molecular-decorators). Our services have different dependencies, which on the one hand are necessary for moleculer (eg "nats"), and on the other hand just happen by chance to be also peerDependencies of moleculer ("etcd3"). Such, yarn berry concludes different virtual moleculer packages.
Until recently, we did not realize it at all. However, probably due to some internal changes in our codebase and dependencies, moleculer now fails to validate objects in their prototype chains because these now originate in different virtual modules!
AFAICT, to solve this, we have to "equalize" dependencies between services by using packageExtensions. It would be really weird, to add dependencies to packages that don't actually use them.
Even more, we need to inject all those dependencies to moleculer-decorators as well.
Finally, because we are using typescript, yarn forces peerDependencies for all (even optional) peerDependencies of dependencies. For instance, moleculer has peerDependencies for debug, pino and winston. So we need to depend on @types/debug and have to inject it into moleculer-decorators as well. Even if one service uses debug, another one pino or a third one winston, all @types/ need to be included. And, since @types/ is a devDependency, we have to use packageExtensions to force that dependency into our common service library which all services depend upon (the devDependency would not be installed otherwise).
The worst thing is, that failing to include any of these dependencies will lead to more than one virtual package and cause runtime errors.

BTW, this only happens with pnp, not node_modules.

How can this be really solved?

PS: I hope, I've been able to describe it clear enough - it's hard to write it out.

[arcanis]

AFAICT, to solve this, we have to "equalize" dependencies between services by using packageExtensions. It would be really weird, to add dependencies to packages that don't actually use them.

If package A depends on package B which has a peer dep on package C, then A uses C. Perhaps not itself, but by picking dependency B they literally promised that they would provide C themselves. The contract must be fulfilled. In theory node_modules also require that, but the node_modules algorithm was hidding this issue by (often) doing the right thing thanks to being able to repeatedly crawl the filesystem. PnP requires one import = one static location, so we cannot rely on such hacks - you have to ensure your dependencies are all properly defined.

I agree it can sometimes be unwieldy when porting old and large codebases. Unfortunately we don't have a "fix everything in one go" tool, as it usually depends on various details of your monorepo. The good news is that once you clean that up and finish the migration, I find it fairly rare to see similar issues come back, since it'll cause errors before you start relying on it everywhere. The main cost is the initial one.

[jeffrson]

Thank you for your quick reply!

I understand that it might be necessary to change how we work with peerDependencies.

Just one single point (which regards yarn interns): is it expected, that I have to put "@types/debug" (for example) into packageExtensions for dependencies, just because I'm using "debug" in a typescript project (apps with libraries)?