[Case Study] One Scope, Multiple Registries

[pickypg]

What package is covered by this investigations?

A private package under the @elastic scope and @elastic/elasticsearch, the Elasticsearch Javascript library.

• @elastic/x private package is published to GitHub's npm package manager.
• @elastic/elasticsearch is public and published to npm (https://www.npmjs.com/package/@elastic/elasticsearch).

Describe the goal of the investigation

Determine how to use one scope with multiple registries like most other build systems.

Investigation report

The problem stems from this line in the npm documentation for registries, which also appears to apply to Yarn, although I could not find it explicitly in Yarn's documentation:

https://docs.npmjs.com/misc/scope#associating-a-scope-with-a-registry

Scopes have a many-to-one relationship with registries: one registry can host multiple scopes, but a scope only ever points to one registry.

(Emphasis mine)

As a high level goal, this approach makes a lot of sense: simplify the lookup process and thus guarantee that once found, it will most likely always be there or error out. However, for complex, internal projects it really breaks down. Internal projects tend to have internal dependencies that will often benefit from public dependencies from the same organization (aka scope).

In other build systems, like Gradle, a series of registries can be defined and the first hit -- when executed in order -- will be used for any given dependency. This is not only helpful, but it seems necessary in the growing world of npm usage where GitHub is now offering a mechanism for private publication alongside npmjs.com (which is perhaps confusing given that GitHub recently acquired npm).

[pickypg]

As a point of interest, Artifactory has "Virtual Registries Repositories" to workaround this limitation.

https://www.jfrog.com/confluence/display/RTF/NPM+Registry

[craigphicks]

https://docs.npmjs.com/misc/scope#associating-a-scope-with-a-registry

Scopes have a many-to-one relationship with registries: one registry can host multiple scopes, but a scope only ever points to one registry.

That is an ambiguous statement. As far as I can tell, what it means is that in any development directory D downstream of a package, P, accessing a scope S, the mapping from S to a register R1 is unique with respect to D. And that mapping is defined by a config file in D, and not a config file in P.

Now suppose that the during testing of P, the scope was mapped to register R0 (not R1).

What it really means then is "but a scope only ever points to one registry as defined by the configuration of the current downstream development directory. It is possible that when P was developed yesterday S was mapped to R0, but since D is using a different configuration file, today in D, S now maps to R1.

It therefore follows that the system design of assigning the mapping from scope to registry is at least not perfect. Of course, almost all the time an unintended scope mapping will just lead to immediate error, and in a rare cases using an incorrect version. However, it is not impossible that it leads to a subtly different and very evil version - if somebody bad is working very hard to leverage that tiny opportunity.

[alethenorio]

Is there any active development in this? Has anybody found a workaround for this issue?

For companies using GitHub, it is quite convenient to be able to push their private packages in the GitHub registry however for the purposes of open source, npm is de-facto place to be. On top of that, it makes sense that companies would name both their organization the same in both places.

So at the moment you're either stuck with offering public packages in GitHub (which isn't great from a user experience PoV) or you are left with 2 options

• Onboard every developer in your company in a paid npm account which can be quite costly depending on the amount of the developers
• Onboard only some of the developers in a paid npm organization which hurts internal development as someone wanting to simply clone a repo and build in order to fix a bug is immediately stopped because they don't have access to the organization npm account.

Has there been any thought of being able to, for example, supply the registry address instead of the scope name?

[wickkidd]

This idea appears to have died in darkness. npm/rfcs#314

I think it was an oversight when creating scopes.

[pickypg]

Has anybody found a workaround for this issue?

#1455 (comment)

If you use a third party tool like Artifactory, then you can join the remote registries (NPM and private GitHub). This works by having your internal project that needs both defining a registry against Artifactory for the designated scope.