Allow different versions in (dev) sections

[basvanberckel]

Is your feature request related to a problem? Please describe.

I would be very helpful if versions are resolved/locked separately for different sections. Currently, the pdm.cli.actions.do_lock procedure locks whatever the highest version between all sections is. pdm.cli.actions.do_sync then looks at the lock file and installs the version it finds there, regardless of the specification given in the section that it is locking for.

pdm.cli.actions.do_lock appears to mention this feature in the TODO at the top of the function, but I could not find a github issue for it.

Describe the solution you'd like

A simplified example pyproject.toml would be

dependencies = [
    "pdm"
]

[tool.pdm.dev-dependencies]
dev = [
    "-e pdm @ git+https://github.com/pdm-project/pdm"
]

pdm install --prod would install the latest version from the package index, while pdm install -d would install the package from the git repo.

[frostming]

It is similar to #46 but with a minor difference

[jjangga0214]

@frostming
I see. Do you have an idea of how to resolve this?

[inigohidalgo]

Hey @frostming, this would be a much welcome feature for my projects as it would allow me to switch between editable installs of dependencies I am developing alongside my project and to specify the package name at runtime when I deploy my code.

I'd be interested in contributing towards this feature. I saw you listed it in #743 as (possible), but it wasn't included in 2.0. Have you made any attempt at resolving this/can you point me towards where I can begin to understand where the problem comes from?

Thanks! :)

[frostming]

@inigohidalgo Thanks for showing interest in this problem. This one is a bit tricky. In the dependency resolver, every requirement has a unique identifier(you can think of unknown numbers x, y, z in an equation to solve) to tell the resolver a solution is needed. If there are multiple requirements for the same package, they should have different identifiers, then how to design the identifier? And what if the child dependencies also have multiple requirements?

A rather brute method is to generate a sub-resolver each time it encounters a "branch". And finally the results from all sub-resolvers get merged to format the lockfile. But this approach apparently slows down the whole dependency resolution, making the situation worse.

And if we are to switch to another resolver, that would be a larger topic.

[BrutalSimplicity]

1. Could you just use the requirement field in this case?
2. Seems the same hashing strategy could be applied to child dependencies?

Forgive me, if this approach is far too naive.

[BrutalSimplicity]

It would be nice if you could tell pdm to create lockfiles based on isolated groups. This way it wouldn't resolve all dependencies by default. I feel like the more common case is the need to create separate lockfiles (or requirements.txt) for different sets of packages.

[andreas-mucha]

I just also want to add that this would be very useful for me. Apart from #1432 which was just merged, this is the only other thing missing for my usual workflow for co-developing different packages.

[BrutalSimplicity]

@frostming Any new thoughts on how to approach this. I've currently taken your naive approach above and run the install in parallel for all combinations of dependency groups specified by the user (i.e. main/local, main/deploy, main/prod), and then generate a requirements file for each configuration that is stored in the repo. This solution allows us to use the correct version of our project dependencies during local development, local deployment, and production release. However, I'm not happy with the solution although it works pretty well for us so far.

A rather brute method is to generate a sub-resolver each time it encounters a "branch". And finally the results from all sub-resolvers get merged to format the lockfile. But this approach apparently slows down the whole dependency resolution, making the situation worse.

Some thoughts:

If this is an opt-in functionality, is there anything wrong with it being slower?

Perhaps multiple processes can be used to help improve performance in this case.

It seems like the need for this is becoming more common as monorepo strategies are becoming more popular again. It would be nice if pdm could support this case, and in particular, the use case for the same editable and local package being able to live side-by-side. Maybe the latter can be a separate feature/use case.

[la-magra]

Definitely something that would be very welcome.

Right now senior dev is preparing our python monorepo to use pdm.

He had to turn every path dependency into an editable and is going to do the production deployments with pdm install --dev --group django-apps, I think.

[project]
dependencies = [
# ...
# "app1 @ file:///${PROJECT_ROOT}/apps/app1"
# ...
# "app12 @ file:///${PROJECT_ROOT}/apps/app12"
]

[tool.pdm.dev-dependencies]
django-apps = [
"-e file:///${PROJECT_ROOT}/apps/app1#egg=app1",
# ...
"-e file:///${PROJECT_ROOT}/apps/app12#egg=app12",
]
testing = [
# ...
]

[frostming]

@la-magra this specific use case should be already solved by passing --no-editable when doing production deployments.

[opticoder]

I'm looking forward for the feature and ready to participate in any testing

[inigohidalgo]

You're a boss fming. Thank you so much for doing this, and I'm sorry I didn't end up participating in the development. I think this feature is the only thing holding me back from using pdm in production, so I am excited to test it. I will pull ad2a66d this week and test it out on my packages.