index should be able to accept multiple URLs

[grepwood]

What's the problem this feature will solve?

I will be able to search all Python repositories available inside organization Intranet. We have a lot.

Describe the solution you'd like

Currently, the index variable in pip.conf only accepts one URL. I would like it to accept multiple URLs, and that the underlying XML-RPC search function considers all specified index URLs for the purpose of searching for packages.

Alternative Solutions

I could ask our Artifactory admins to make a virtual repository, but those are not compliant as per organization policies. Everyone abused virtual repositories, it was difficult to untagle who depends on what, and now they're simply not allowed. This is a losing fight to pick.

Additional context

I understand that XML-RPC search has been historically abused against pypi.org by unresponsive actors. This feature request does not concern pypi.org and is not going to be used for searching for things residing at pypi.org, but instead for local repositories inside an organization.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

You probably just want extra-index-url.

[grepwood]

extra-index-url is for 3rd party repositories from which you can install, not search.
Relevant docs:

• https://pip.pypa.io/en/stable/cli/pip_search
• https://pip.pypa.io/en/stable/cli/pip_install

[notatallshaw]

Yeah, for this to be added the path forward would be for pip search to support extra-index-url.

However as XML-RPC search API is not based on a Python packaging standard it is unlikely to ever get any new features, and is far more likely to be deprecated and removed from pip at some point.

[grepwood]

That doesn't make any sense. pip search doesn't even take index-url into account. It needs a separate index variable in pip.conf to work.

I'll just write a PR.

[pfmoore]

As @notatallshaw noted, we almost certainly won't accept new features for pip search, as it relies on the non-standard XML-RPC API, which PyPI no longer supports (even if 3rd party indexes do).

If you want to spend time writing a PR, that's fine, but please be aware that it will likely be wasted effort. You would probably be better writing your own "index search" utility, with whatever features you want, instead of trying to get the functionality included in pip. The pip code is open source, so feel free to copy the implementation of pip search as a starting point.

[notatallshaw]

pip search doesn't even take index-url into account. It needs a separate index variable in pip.conf to work.

Ah, I forgot that detail as it's been many years since I used it. If pip search was still well supported it would still make sense for me to model indexl-url & extra-index-url with an extra-index so that users don't find changes to index to be breaking. That said:

I'll just write a PR.

Like @pfmoore I would be very skeptical of accepting a PR to modify pip search in any way other than deprecating it. Especially given the xmlrpc.client is known to not be secure.

[grepwood]

I see. Thank you very much for your honesty. I've a question, since both of you may very well know the answer, perhaps. And that is going to have an influence on what I'd like to cook. I did not elaborate why it doesn't make sense that pip search would consider extra-index-url. I've encountered environments where index-url does not point to a PyPI mirror - instead, this place is taken by an internal 3rd party repo. A PyPI mirror is instead put into extra-index-url. It sounds wrong to me, but it worked in those environments. Should pip be even used like this, that PyPI or its mirror is in extra-index-url?

The reason why I'm asking, is because if we assume extra-index-url may only hold XML-RPC capable mirrors, then it's obviously going to break not only in this kind of environment I've described, but also in environments that have 3rd party repos that do not support XML-RPC. This is a precaution that I've had in mind that made me go for expanding upon index in the first place. If this is instead used as a parameter that can only hold capable mirrors, then it's not going to trespass on mirrors that don't do that kind of thing.

[notatallshaw]

I've encountered environments where index-url does not point to a PyPI mirror - instead, this place is taken by an internal 3rd party repo. A PyPI mirror is instead put into extra-index-url. It sounds wrong to me, but it worked in those environments. Should pip be even used like this, that PyPI or its mirror is in extra-index-url?

This is common, but unfortunately has security implications because, among other issues, pip does not guarantee the order in which the indexes are read.

is because if we assume extra-index-url may only hold XML-RPC capable mirrors, then it's obviously going to break not only in this kind of environment I've described, but also in environments that have 3rd party repos that do not support XML-RPC.

I agree with your original statement , what I meant in my suggestion is that if pip search were well supported (it's not), it makes sense to syntactically mirror extra-index-url with extra-index in the same way index-url is syntactically mirrored with index.

This is a precaution that I've had in mind that made me go for expanding upon index in the first place.

My concern is that this API is not well supported and making any change to how it's processed may end up unintentionally breaking other people's workflows. It’s another reason why any PR is going to be difficult to justify.