Chooses explicit source where it should choose primary

[black-snow]

Description

Poetry seems to install dependencies from groups I explicitly exclude and then fails because there's no candidate for my platform.
Poetry seems to select the explicit source instead of the primary where it should not.

I'm on macOS and apparently there's no suitable CPU-only version for torch = "^2.3.0" at https://download.pytorch.org/whl/cpu.

So what I did:

• move torch to dev dependencies (the full CUDA version)
• move torch CPU variant to an extra group inference
• add pypi as primary source
• add https://download.pytorch.org/whl/cpu as explicit source

[tool.poetry.group.dev.dependencies]
torch = "^2.3.0"

[tool.poetry.group.inference.dependencies]
torch = { version = "^2.3.0", source = "pytorch-cpu-only" }

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

[[tool.poetry.source]]
name = "pytorch-cpu-only"
url = "https://download.pytorch.org/whl/cpu"
priority = "explicit"

Now I'd expect poetry install --without inference to ... work. But it doesn't. It's looking for torch 2.4.1+cpu, not finding any candidate, and thus fails.

I also tried marking the cpu-only source as supplemental and optional but to no avail.

I see perhaps related issues but none seems to really match.

Workarounds

nope

Poetry Installation Method

homebrew

Operating System

macOS 14.7

Poetry Version

1.8.3

Poetry Configuration

cache-dir = "/Users/xxx/Library/Caches/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
repositories.pytorch-cpu-only.url = "https://download.pytorch.org/whl/cpu"
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/xxx/Library/Caches/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

see above

Poetry Runtime Logs

Loading configuration file /Users/xxx/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/xxx/Library/Application Support/pypoetry/auth.toml
Adding repository PyPI (https://pypi.org/simple/) and setting it as primary
Adding repository pytorch-cpu-only (https://download.pytorch.org/whl/cpu) and setting it as explicit
Using virtualenv: /Users/xxx/projects/xxx/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 1 update, 0 removals, 69 skipped

  - Installing annotated-types (0.7.0): Pending...
  - Installing annotated-types (0.7.0): Skipped for the following reason: Already installed
  ...
  - Updating torch (2.4.1 -> 2.4.1+cpu): Pending...
Checking if keyring is available
  ...
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
  - Updating torch (2.4.1 -> 2.4.1+cpu): Failed

  Stack trace:

  6  /opt/homebrew/Cellar/poetry/1.8.3_2/libexec/lib/python3.12/site-packages/poetry/installation/executor.py:285 in _execute_operation
      283│ 
      284│             try:
    → 285│                 result = self._do_execute_operation(operation)
      286│             except EnvCommandError as e:
      287│                 if e.e.returncode == -2:

  5  /opt/homebrew/Cellar/poetry/1.8.3_2/libexec/lib/python3.12/site-packages/poetry/installation/executor.py:395 in _do_execute_operation
      393│             return 0
      394│ 
    → 395│         result: int = getattr(self, f"_execute_{method}")(operation)
      396│ 
      397│         if result != 0:

  4  /opt/homebrew/Cellar/poetry/1.8.3_2/libexec/lib/python3.12/site-packages/poetry/installation/executor.py:527 in _execute_update
      525│ 
      526│     def _execute_update(self, operation: Install | Update) -> int:
    → 527│         status_code = self._update(operation)
      528│ 
      529│         self._save_url_reference(operation)

  3  /opt/homebrew/Cellar/poetry/1.8.3_2/libexec/lib/python3.12/site-packages/poetry/installation/executor.py:586 in _update
      584│ 
      585│     def _update(self, operation: Install | Update) -> int:
    → 586│         return self._install(operation)
      587│ 
      588│     def _remove(self, package: Package) -> int:

  2  /opt/homebrew/Cellar/poetry/1.8.3_2/libexec/lib/python3.12/site-packages/poetry/installation/executor.py:558 in _install
      556│             archive = self._download_link(operation, Link(package.source_url))
      557│         else:
    → 558│             archive = self._download(operation)
      559│ 
      560│         operation_message = self.get_operation_message(operation)

  1  /opt/homebrew/Cellar/poetry/1.8.3_2/libexec/lib/python3.12/site-packages/poetry/installation/executor.py:736 in _download
      734│ 
      735│     def _download(self, operation: Install | Update) -> Path:
    → 736│         link = self._chooser.choose_for(operation.package)
      737│ 
      738│         if link.yanked:

  RuntimeError

  Unable to find installation candidates for torch (2.4.1+cpu)

  at /opt/homebrew/Cellar/poetry/1.8.3_2/libexec/lib/python3.12/site-packages/poetry/installation/chooser.py:74 in choose_for
       70│ 
       71│             links.append(link)
       72│ 
       73│         if not links:
    →  74│             raise RuntimeError(f"Unable to find installation candidates for {package}")
       75│ 
       76│         # Get the best link
       77│         chosen = max(links, key=lambda link: self._sort_key(package, link))
       78│ 

Cannot install torch.

[black-snow]

Btw, --only yields the same result - doesn't seem to be specific to --without.

When I comment the inference group everything is fine.

[dimbleby]

you have included torch in two groups. Excluding only one of them does not remove the requirement to install torch.

[black-snow]

Yes, but the one in dev should be taken from the primary source and thus install the cuda-powered one from pypi which does have a version for macOS. But poetry is picking of the explicit source.
@dimbleby

/edit: Yep, so the issue might be a bit misphrased.
/edit2: Rephrased

[dimbleby]

the locking process chooses a single solution compatible with all groups

there's an issue out there requesting support for mutually exclusive groups, you can close this as a duplicate of that

[black-snow]

Refs:

• Instructions for installing PyTorch #6409
• Support for mutually exclusive dependencies across groups (e.g. path dependencies during development only) #1168

Thanks for looking into this @dimbleby

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.