how to install with version annotation and wheel? (e.g. pytorch cpu version) #694

jjangga0214 · 2021-10-21T18:07:27Z

Pytorch's official page guides user to install the cpu-only mode of package by the following.

pip3 install torch==1.10.0+cpu \ 
  torchvision==0.11.0+cpu \ 
  torchaudio==0.10.0+cpu \
  -f https://download.pytorch.org/whl/cpu/torch_stable.html

If I run

pdm add torch~=1.10.0+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html

then an error occurs.

Usage: pdm [OPTIONS] [COMMANDS] ...
pdm: error: unrecognized arguments: -f https://download.pytorch.org/whl/cpu/torch_stable.html

If I run

pdm add torch~=1.10.0+cpu

then another error occurs.

[RequirementError]: The local path torch=1.10.0+cpu does not exist.
Add '-v' to see the detailed traceback

How can I do?

Thanks.

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
➜  pdm (☸ |k3d-ateam:local) pdm info && pdm info --env
PDM version:        1.9.0                                                 
Python Interpreter: /home/jjangga/.pyenv/versions/3.9.7/bin/python3 (3.9) 
Project Root:       /media/jjangga/SHARE/experiment/pdm                   
Project Packages:   /media/jjangga/SHARE/experiment/pdm/__pypackages__/3.9
{
  "implementation_name": "cpython",
  "implementation_version": "3.9.7",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "5.11.0-37-generic",
  "platform_system": "Linux",
  "platform_version": "#41~20.04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021",
  "python_full_version": "3.9.7",
  "platform_python_implementation": "CPython",
  "python_version": "3.9",
  "sys_platform": "linux"
}

frostming · 2021-10-22T06:53:53Z

pdm currently doesn't support find link pages with -f url. As a workaround, you can use the absolute URL for the specific package: pdm add 'torch @ https://url/to/torch.whl'

jjangga0214 · 2021-10-22T10:03:33Z

@frostming Thanks!
But there're many platform-specific wheels.

And my team members use different operating systems.

Can pdm cover this case declaratively?
I want pdm install to just work automatically in every teammates computer.'
For example, in macOS, wheels are not needed, unlike in Linux.

Should I workaround it by group?

frostming · 2021-10-22T10:25:21Z

I think you may need #696 which just landed in the main branch. Just that torch~=1.10.0+cpu isn't a valid req string on both pdm and pip. torch==1.10.0+cpu instead

jjangga0214 · 2021-10-22T12:15:24Z

@frostming Oh, great!
Would it be available as a new version soon?

frostming · 2021-10-22T14:39:36Z

yes, but would you like to help testing if possible?

jjangga0214 · 2021-10-22T14:55:09Z

@frostming Sure, how?

frostming · 2021-10-22T15:19:54Z

install the dev version and test if it works

should be easy to do with pip install git+https://github.com/pdm-project/pdm.git@main#egg=pdm

jjangga0214 · 2021-10-23T08:38:36Z

@frostming
Thanks, I installed it. Then how can I write the pyproject.toml?

I tried something like an above example, but failed.

frostming · 2021-10-23T09:01:09Z

use == please

jjangga0214 · 2021-10-23T09:50:36Z

@frostming
Yeah thanks!

dependencies = [
  "torch==1.10.0+cpu",
]

But an error occurs.

root@ccd295b0870b:/experiment# pdm install
Traceback (most recent call last):
  File "/usr/local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 200, in main
    return Core().main(args)
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 138, in main
    self.ensure_project(options, obj)
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 108, in ensure_project
    migrate_pyproject(options.project)
  File "/usr/local/lib/python3.9/site-packages/pdm/cli/actions.py", line 677, in migrate_pyproject
    if project.pyproject and "project" in project.pyproject:
  File "/usr/local/lib/python3.9/site-packages/pdm/project/core.py", line 99, in pyproject
    data = atoml.parse(self.pyproject_file.read_text("utf-8"))
  File "/usr/local/lib/python3.9/site-packages/atoml/api.py", line 68, in parse
    return Parser(string).parse()
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 152, in parse
    key, value = self._parse_table()
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 1159, in _parse_table
    key_next, table_next = self._parse_table(name, table)
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 1146, in _parse_table
    item = self._parse_item()
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 298, in _parse_item
    return self._parse_key_value(True)
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 389, in _parse_key_value
    val = self._parse_value()
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 561, in _parse_value
    return self._parse_array()
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 710, in _parse_array
    elems.append(self._parse_value())
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 563, in _parse_value
    return self._parse_inline_table()
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 773, in _parse_inline_table
    key, val = self._parse_key_value(False)
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 370, in _parse_key_value
    key = self._parse_key()
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 414, in _parse_key
    return self._parse_bare_key()
  File "/usr/local/lib/python3.9/site-packages/atoml/parser.py", line 466, in _parse_bare_key
    raise self.parse_error(ParseError, "Empty key found")
atoml.exceptions.ParseError: Empty key found at line 25 col 3

I still doubt if this is right setting.

frostming · 2021-10-23T09:53:37Z

It is TOML, not JSON:

[[tool.pdm.source]]
type = "find_links"
url = "..."
name = "torch"

jjangga0214 · 2021-10-23T10:03:22Z

@frostming
Ah, I see.

Hmm still having an error.

pyproject.toml

[project]
name = ""
version = ""
description = ""
authors = [
    {name = "", email = ""},
]
dependencies = [
  "torch==1.10.0+cpu",
]
requires-python = ">=3.9"
dynamic = ["classifiers"]
license = {text = "MIT"}

[project.urls]
homepage = ""

[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"

[[tool.pdm.source]]
type = "find_links"
url = "https://download.pytorch.org/whl/cpu/torch_stable.html"
name = "torch"

installation

root@ccd295b0870b:/experiment# pdm install -v
Lock file does not exist or is incompatible, trying to generate one...
======== Start resolving requirements ========
  torch==1.10.0+cpu
  python>=3.9
  Adding requirement torch==1.10.0+cpu
	Could not find any matching candidates.
	Could not find any matching candidates even when considering pre-releases.
	Found but non-matching candidates:
	  <Candidate torch 1.10.0 from https://pypi.org/simple/torch/>
	  <Candidate torch 1.9.1 from https://pypi.org/simple/torch/>
	  <Candidate torch 1.9.0 from https://pypi.org/simple/torch/>
	  <Candidate torch 1.8.1 from https://pypi.org/simple/torch/>
	  <Candidate torch 1.8.0 from https://pypi.org/simple/torch/>
	  <Candidate torch 1.7.1 from https://pypi.org/simple/torch/>
	  <Candidate torch 1.7.0 from https://pypi.org/simple/torch/>
	  <Candidate torch 1.6.0 from https://pypi.org/simple/torch/>
	  <Candidate torch 1.5.1 from https://pypi.org/simple/torch/>
	  <Candidate torch 1.5.0 from https://pypi.org/simple/torch/>
	  ... [224 more candidate(s)]
🔒 Lock failed
Unable to find a resolution that satisfies the following requirements:
  torch==1.10.0+cpu (from project)
Please make sure the package names are correct. If so, you can either loosen the version constraints of these dependencies, or set a narrower `requires-python` range in the pyproject.toml.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 341, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 173, in _add_to_criteria
    raise RequirementsConflicted(criterion)
resolvelib.resolvers.RequirementsConflicted: Requirements conflict: NamedRequirement(name='torch', marker=None, extras=(), specifier=<SpecifierSet('==1.10.0+cpu')>, editable=False)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 200, in main
    return Core().main(args)
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 153, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 148, in main
    f(options.project, options)
  File "/usr/local/lib/python3.9/site-packages/pdm/cli/commands/install.py", line 38, in handle
    actions.do_lock(project, strategy="all")
  File "/usr/local/lib/python3.9/site-packages/pdm/cli/actions.py", line 68, in do_lock
    mapping, dependencies = resolve(
  File "/usr/local/lib/python3.9/site-packages/pdm/resolver/core.py", line 31, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 343, in resolve
    raise ResolutionImpossible(e.criterion.information)
resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=NamedRequirement(name='torch', marker=None, extras=(), specifier=<SpecifierSet('==1.10.0+cpu')>, editable=False), parent=None)]

info

root@ccd295b0870b:/experiment# pdm info
PDM version:        1.9.1.dev13+gf252992
Python Interpreter: /usr/local/bin/python (3.9)
Project Root:       /experiment
Project Packages:   /experiment/__pypackages__/3.9

frostming · 2021-10-23T10:26:55Z

Oh, I've updated the main branch, would you try again?

jjangga0214 · 2021-10-23T10:54:07Z

@frostming I tried!
And it works!
Yeah~!

So there's only one issue left.
As I've mentioned earlier, pytorch should be installed in different ways, depending on the target.

Scenario 1. different version by environment:
- development environment: it should be torch==1.10.0 (gpu+cpu version), as training deep learning model needs both gpu and cpu.(unless it's a really small model)
- production environment: it should be torch==1.10.0+cpu (cpu only version), as inference of deep learning model does not need gpu. (unless it's a really big model)
Scenario 2. different version by platform:

Even if we decide to always use cpu-only version no matter if the environment is for development or production, pytorch should be installed differently by platform.
For example, the official guides says we should use torch==1.10.0 for macOS, but torch==1.10.0+cpu for Linux.

Currently I am thinking using concepts of group and optional dependency for solving this issue.

pyproject.yaml

[project.optional-dependencies]
cpu = [
  "torch==1.10.0+cpu",
]
gpu = [
  "torch==1.10.0"
]

But in this way, we should execute the command differently (not just simply pdm install as we should care group), and torch is actually not optional, but specified as optional..

Would it be the best solution?
If not, could you suggest another opinion?

frostming · 2021-10-23T11:27:39Z

This is a non-trivial feature and not supported yet: #46 #564 . So for now you can only have one torch version locked in the lock file.

jjangga0214 · 2021-10-23T14:28:27Z

(Thanks! It was great to know find_links become available from the next version!)

lsmith77 · 2023-03-29T13:39:57Z

I am using PDM, version 2.4.8 on OSX and I am trying to get the CPU verson of PyTorch installed.

I am trying

pdm add "torch>=2.0.0+cpu"
Adding packages to default dependencies: file:///${PROJECT_ROOT}/torch%3E%3D2.0.0%2Bcpu
See /var/folders/61/02k_wd1j7f57mhzbflsv1s2m0000gn/T/pdm-lock-hn01l8j6.log for detailed debug log.
[FileNotFoundError]: [Errno 2] No such file or directory: '/Users/lsmith/htdocs/nlp_api-flexible/torch>=2.0.0+cpu'
Add '-v' to see the detailed traceback

Without the +cpu it manages to install torch but then I don't have the CPU version, which means it tries to load CUDA related libraries.

lsmith77 · 2023-03-29T14:24:39Z

the following seems to have done the trick

dependencies = [
    "torch>=2.0.0",
]

[[tool.pdm.source]]
type = "find_links"
url = "https://download.pytorch.org/whl/cpu/torch_stable.html"
name = "torch"

my pdm.lock looks like this:

[[package]]
name = "torch"
version = "2.0.0+cpu"
requires_python = ">=3.8.0"
summary = "Tensors and Dynamic neural networks in Python with strong GPU acceleration"
dependencies = [
    "filelock",
    "jinja2",
    "networkx",
    "sympy",
    "typing-extensions",
]

frostming · 2023-03-29T14:32:24Z

Without the +cpu it manages to install torch

You are making a mistake: you thought >= works similarly to == and should work when combined with 2.0.0+cpu. However this is wrong, +cpu local version can't be used with comparison operators but can be used with equality and inequality operators.

zagorulkinde · 2024-10-03T10:41:47Z

the following seems to have done the trick

dependencies = [
    "torch>=2.0.0",
]

[[tool.pdm.source]]
type = "find_links"
url = "https://download.pytorch.org/whl/cpu/torch_stable.html"
name = "torch"

my pdm.lock looks like this:

[[package]]
name = "torch"
version = "2.0.0+cpu"
requires_python = ">=3.8.0"
summary = "Tensors and Dynamic neural networks in Python with strong GPU acceleration"
dependencies = [
    "filelock",
    "jinja2",
    "networkx",
    "sympy",
    "typing-extensions",
]

Thanks, works like a charm!

jjangga0214 added the 🐛 bug Something isn't working label Oct 21, 2021

jjangga0214 mentioned this issue Oct 21, 2021

is it possible installing with wheel and version annotation? David-OConnor/pyflow#149

Open

jjangga0214 closed this as completed Oct 23, 2021

pikeas mentioned this issue May 24, 2022

Unable to install Pytorch (M1 Mac prerelease) #1093

Closed

1 task

ValerianRey mentioned this issue Aug 25, 2024

In testing actions install without CUDA. TorchJD/torchjd#47

Closed

afterow mentioned this issue Nov 1, 2024

Can you add support for pdm? Refound-445/nonebot-plugin-nailongremove#18

Closed

jantrienes mentioned this issue Nov 12, 2024

Please offer <version>+cpu-versioned packages for MacOS and linux ARM CPU-only builds pytorch/pytorch#136275

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how to install with version annotation and wheel? (e.g. pytorch cpu version) #694

how to install with version annotation and wheel? (e.g. pytorch cpu version) #694

jjangga0214 commented Oct 21, 2021

frostming commented Oct 22, 2021 •

edited

Loading

jjangga0214 commented Oct 22, 2021 •

edited

Loading

frostming commented Oct 22, 2021 •

edited

Loading

jjangga0214 commented Oct 22, 2021

frostming commented Oct 22, 2021

jjangga0214 commented Oct 22, 2021

frostming commented Oct 22, 2021 •

edited

Loading

jjangga0214 commented Oct 23, 2021 •

edited

Loading

frostming commented Oct 23, 2021

jjangga0214 commented Oct 23, 2021 •

edited

Loading

frostming commented Oct 23, 2021

jjangga0214 commented Oct 23, 2021 •

edited

Loading

frostming commented Oct 23, 2021

jjangga0214 commented Oct 23, 2021 •

edited

Loading

frostming commented Oct 23, 2021

jjangga0214 commented Oct 23, 2021 •

edited

Loading

lsmith77 commented Mar 29, 2023

lsmith77 commented Mar 29, 2023

frostming commented Mar 29, 2023

zagorulkinde commented Oct 3, 2024

how to install with version annotation and wheel? (e.g. pytorch cpu version) #694

how to install with version annotation and wheel? (e.g. pytorch cpu version) #694

Comments

jjangga0214 commented Oct 21, 2021

Environment Information

frostming commented Oct 22, 2021 • edited Loading

jjangga0214 commented Oct 22, 2021 • edited Loading

frostming commented Oct 22, 2021 • edited Loading

jjangga0214 commented Oct 22, 2021

frostming commented Oct 22, 2021

jjangga0214 commented Oct 22, 2021

frostming commented Oct 22, 2021 • edited Loading

jjangga0214 commented Oct 23, 2021 • edited Loading

frostming commented Oct 23, 2021

jjangga0214 commented Oct 23, 2021 • edited Loading

frostming commented Oct 23, 2021

jjangga0214 commented Oct 23, 2021 • edited Loading

frostming commented Oct 23, 2021

jjangga0214 commented Oct 23, 2021 • edited Loading

frostming commented Oct 23, 2021

jjangga0214 commented Oct 23, 2021 • edited Loading

lsmith77 commented Mar 29, 2023

lsmith77 commented Mar 29, 2023

frostming commented Mar 29, 2023

zagorulkinde commented Oct 3, 2024

frostming commented Oct 22, 2021 •

edited

Loading

jjangga0214 commented Oct 22, 2021 •

edited

Loading

frostming commented Oct 22, 2021 •

edited

Loading

frostming commented Oct 22, 2021 •

edited

Loading

jjangga0214 commented Oct 23, 2021 •

edited

Loading

jjangga0214 commented Oct 23, 2021 •

edited

Loading

jjangga0214 commented Oct 23, 2021 •

edited

Loading

jjangga0214 commented Oct 23, 2021 •

edited

Loading

jjangga0214 commented Oct 23, 2021 •

edited

Loading