Alternative package sources and the source property of individual dependencies

[ggabriel96]

Hi there!

The PR #908 improved the support for alternative package indexes. However, AFAIK, not everything is currently documented. In particular, I could not find docs about the feature of specifying a source for specific dependencies. It is also not very clear to me what happens when a project has multiple repositories configured.

IIUC, from the docs:

• adding a repository will make Poetry search for packages in it, giving it precedence over PyPI
• setting the default attribute of a repository to true will disable PyPI
• setting the secondary attribute of a repository to true will keep PyPI as the main repository

However, what does precendence over PyPI mean? As explained briefly in the warning in item 10 here, doing something like this:

python -m pip install --extra-index-url http://my.package.repo/simple SomePackage

has some problems:

Using this option to search for packages which are not in the main repository (such as private packages) is unsafe, per a security vulnerability called dependency confusion: an attacker can claim the package on the public repository in a way that will ensure it gets chosen over the private package.

So, when simply adding a repository repo without setting default or secondary (or having both be false), is it possible for a package to have a better match in PyPI than in repo and get installed from there? And when I specifically set the source of a package to be repo, will PyPI ever be considered? And if repo is marked as secondary?

Having these questions answered, I volunteer to update the documentation.
By the way, much love for Poetry ❤️

---

Edit 1: apparently, the source option for a dependency makes it the only source that will be considered:

https://github.com/python-poetry/poetry/blob/1.1/poetry/json/schemas/poetry-schema.json#L269-L272

Is that correct?

---

Edit 2: so I've searched through Poetry tests and found some answers. IIUC:

• when multiple repositories are present (and none were marked as secondary or default), Poetry chooses the one that provides the best match (is most recent) for the package
• if an explicit source is specified for a dependency, it is indeed used, even if the repository is marked as secondary. I don't fully understand the test with transitive dependencies yet
• if a repository is marked as secondary, it won't be chosen as the source of a package

Questions remain, though:

• if a dependency has an explicit source, but that source for some reason does not have that package, will it fall back to other repositories or fail with an error?
• when exactly is a secondary repository used? I guess when it is the only repo that provides a dependency? And when it has a better match for a package, will it be used?

---

Edit 3: just found this PR: #3400. So only Poetry's behavior is left to be better explained.

[lainisourgod]

Thanks, @ggabriel96 for your research! I was confused by the docs too.

Maybe you can anwer my question related to private repos.

• If poetry lock was run against public PyPI
• then pyproject.toml has been changed to use only private repo
• and then poetry install was run against new pyproject.toml

will poetry install dependencies from private repo or public?

My usecase is disabling private repo during development because it's slow: #4113

[ggabriel96]

That's a very good question. I have not tried those specific steps, I would need to investigate it. I think it comes down to what is saved in the lockfile. For example, if your private repo has public PyPI as upstream and all hashes match, it could work through the private repo. But I don't know if the source repo is saved to the lockfile too and that mismatch would cause a failure.

By the way, I also faced the same problem as in your issue. Just adding a single package from a private source made dependency resolution jump from around 20s to something like 150s. I suspect Poetry is searching for every dependency in the private repo before falling back to PyPI, since the docs mention that any custom repo is given priority (in my case, I didn't set PyPI as upstream). However, I don't remember if setting it to secondary solves that (I don't think so, so there could be a bug).

[sneakers-the-rat]

This is a persistent issue that has, I think, more than a few parts to it. I'm going to bump this thread because I think it's maybe the most relevant/best researched of those I've seen (thanks @ggabriel96 ). I think there are a bunch of separate issues that might be helped by treating them altogether

I haven't been all the way through the code, but I think the root of this is that there need to be more conditions considered when a Package is considered "the same" as another in addition to version. There are a few downstream problems and possible solutions, mostly having to do with the handling of secondary sources and, I've tried to gather some representative issues of each.

• Wheels built for different architectures that come from different repositories aren't respected, eg. when I use piwheels.org that builds wheels for armv6 and armv7 as a primary source, poetry fails to install on x86_64 systems. eg. Multi-architecture wheel selection across disperate sources (with markers) #5205 , Poetry fails to check package from PyPI if it was found for another arch on a private repo #5227 This also seems to be true for different architectures that are on the same repository: Poetry doesn't check extra sources for package #4854
• More broadly, there isn't a way of specifying the architecture that the package is to be installed on. It's possible to specify different package versions using platform tags, but specifying that the same version should be gotten from different repositories are treated the same, ie. the following is marked as being the same and the dual dependency is not respected:

[tool.poetry.dependencies]
package = [
  {version="^1.0.0", markers="machine==armvl7", source="piwheels"},
  {version="^1.0.0", markers="machine!=armvl7", source="pypi"}
]

I think this is related to this too: #5150

• The precedence and specification of sources is not altogether clear, as noted in this thread and in some of these issues. Particularly, the secondary option seems to be broken in some instances.: repository.secondary=true + dependency.source broken? #5122 Adding a source even with secondary true take precedence #4704 Poetry doesn't try public pypi when private pypi included #3855 Source key word for Private repository #4920
• Since there isn't a clear condition when a secondary source is to be consulted, there are significant perf costs, especially when large packages or slow repos are specified. If all output requirements are satisfied by a first hit, then the secondary sources shouldn't be consulted. If a hit is found on the first repo, but one that doesn't satisfy all constraints including architecture/OS/etc., then the secondary source should be consulted, and so on. eg.: Dependency resolution much slower when adding secondary sources #5096

Some Initial Ideas of How to Fix

• Specify architectures that are to be supported by a package, try to find and keep wheels for each supported architecture (or allow building from source when no wheel is found.
• Another way might be to support multiple sources in the Package object from poetry-core. That might be generalized to keep separate versions that vary along any of the possible dimensions? This seems like something that must be being done somewhere, but as far as I can from the debugging output they're often collapsed.
• Mark all entries in the lockfile as having a source, rather than PyPI being a special case with no source
• Numerical priority for sources, rather than primary, default, and secondary, which have some tricky definitions, at least in my reading of the docs. This would let you set a chain of sources to check, and given more means of specifying when a particular dependency is filled mentioned above (eg. arch, os, etc.) then the number of hits to each could be minimized. It also seems like it should be possible to specify package markers in sources, eg. saying that one repository will supply most of the armv7 requirements so that in the case that there >2 repositories poetry could be smart about choosing which to ping? Related to: Poetry install dependency doesnt work when we use two private repository #4900
• Clearer precedence hierarchy of how sources are consulted. Eg being able to specify a global repository like Install dependencies from private repository based on global configuration #4817 but also make overrides at each level clearer (when does secondary > source)

In the meantime, it seems like, since url-based dependencies are treated differently in several places, it seems like doing that rather than using the repository fixes a lot of issues, eg. #4659 and #4439 and

I got started on one hacky part of the solution here (preserving filenames and hashes that can be used in the pip install stage):
#5205 (comment)

Anyway yes just trying to focus some work so we can pull a PR together!!! Looking for direction on how best to tackle this problem.