[Case Study] [webpack] How to resolve virtual packages using a berry api

[edeustace]

What package is covered by this investigations?

webpack@^5

Describe the goal of the investigation

Find out how best to resolve virtual packages with different paths to the same package so that webpack can make a reference to a linked library using the DllReferencePlugin.

Investigation report
See this bug report in webpack: webpack/webpack#10375

In a nutshell: In order for the DllReferencePlugin to make a reference to a prebuilt bundle in a library build with webpack and DllPlugin, the reference plugin needs to match the library package with the local package. At the moment this is doing a simple string match. This won't work for virtual packages because the paths will be different, so we'd need to do some work resolving both paths to see if they are the same yarn package. I'm guessing the logic for this is in berry, any tips on how to use would be appreciated.

[arcanis]

Various interesting points:

1. At its core, it's because you're listing react-dom as a regular dependency of both dll-lib and lib-user. Ideally I think dll-lib would have a peer dependency on react-dom and react (I guess? I don't know exactly what the DLL plugin does with peer dependencies). This would ensure that the version used by dll-lib and lib-user is always exactly the same (so Yarn would generate a single virtual path, but cf point 3).

2. To top it off you're using different ranges for the react and react-dom dependencies: dll-lib uses ^16.12.0 and lib-user uses ^16.8.1. There is opportunity for them to be locked to completely different versions! If they were the same range Yarn would aggregate them into the exact same reference, which would in turn be optimized into a single virtual path (since they share the same dependency set) ... if not for point 3.

3. To top it off, dll-lib and lib-user aren't workspaces part of the same project. Yarn creates the virtual paths unique to the local project without expecting them to be consistent with other unrelated projects, hence why the virtual paths would be different even if the exact same version was used. You could workaround this by giving the same name to both packages, but it would be kind of a hack.

My suggestion:

• Turn react and react-dom into peer dependencies in dll-lib
• Move dll-lib and lib-user into workspaces of a same project
• Use a constraint if you have multiple lib-user to ensure they all use the exact same ranges for react and react-dom.

[arcanis]

Reading a bit more how the DLL work, peer dependencies wouldn't be a solution because dll-lib will be bundled independently from its consumers (so it won't have access to any peer dep).

I guess in this case you'll have to rely on the fact that Yarn will always resolve a same range to the same version (this makes the constraint even more important to be sure they don't accidentally desynchronize), and will optimize away duplicate dependency sets when possible.

[edeustace]

thanks @arcanis, re point 2 (different versions) - that was a mistake - I've synched them but the original issue still exists.

I'd prefer not to use workspaces as I would like the dll-lib to be built independently.

Is there no mechanism I could use to figure out that 2 differing virtual paths are pointing to the same package?

[arcanis]

If you don't use workspaces and the two projects have the same "name" field they will share the same virtual path I think. But that's one more assumption 🙂

[edeustace]

Just tested that ^ -yes it's true - the virtual paths match and the Dll linking works. However as you said it's a bit of a hack.

So yarn doesn't resolve these virtual paths at all? Or it does but it's not easily accessible for external use?

[sokra]

I curious which factors are used to generate the hash of the virtual package path? Why is the name of the root package one of them?

[arcanis]

Technically we use the locator of the package to generate the virtual path that will be used by its child dependencies. For workspaces the locator has the name of the package, and its reference is the relative path from the project root.

We could potentially special-case workspaces to only generate their virtual path based on the relative path, but I'm not sure how much of a good idea it would be ... 🤔