Add support for listing HTTP cached packages in pip cache list

[a-sajjad72]

This PR adds support for listing HTTP-cached packages in pip cache list and introduces new flags to control output, addressing issue #10460.

Problem

Currently, pip cache list only shows locally built wheels stored in the wheels/ cache directory, but ignores HTTP cached packages stored in the http-v2/ or http directory. This leads to confusing behavior where users see "No locally built wheels cached" even when pip has cached wheel files from PyPI downloads.

$ pip cache info
Package index page cache size: 89 MB
Number of HTTP files: 815
Locally built wheels size: 8.9 MB
Number of locally built wheels: 35

$ pip cache list
No locally built wheels cached.  # Misleading - there are cached packages!

Solution

This PR extends pip cache list to extract package information from cached file content by inspecting the package structure offline, and adds flags to control what is listed:

1. Parse CacheControl metadata using cachecontrol.Serializer to locate cached response details.
2. Inspect cached body files to derive artifact filenames and sizes:
   • Wheel files: Read .dist-info/WHEEL metadata from the ZIP; when needed, construct a filename using the first Tag: entry; fall back to {name}-{version}.whl.
   • Tarballs: Read the tar structure to derive {name}-{version}.tar.gz from the root directory name.
3. Add CLI flags:
   • --http: list only HTTP cache files.
   • --all: list both locally built wheels and HTTP cache files in a unified list, suffixing HTTP entries with [HTTP cached].
4. Operates completely offline (no network), and is index-agnostic (not PyPI-specific).

CLI changes

Users can control which cache types to list:

pip cache list            # List locally built wheels (default; unchanged)
pip cache list --http     # List only HTTP cache files
pip cache list --all      # Unified list; HTTP entries marked with [HTTP cached]

Examples

Human-readable output shows filenames and sizes. When --http is used alone, entries are listed under an HTTP section; with --all, entries are unified and HTTP items are suffixed.

$ pip cache list --http
HTTP cache files:
 - certifi-2025.1.31.tar.gz (167 kB)
 - Django-2.1.15-py3-none-any.whl (7.3 MB)

$ pip cache list --all
 - certifi-2025.1.31.tar.gz (167 kB) [HTTP cached]
 - Django-2.1.15-py3-none-any.whl (7.3 MB) [HTTP cached]
 - setuptools-68.0.0-py3-none-any.whl (804 kB)

Implementation Details

The implementation extracts filenames by reading cached package structures:

Wheel Files

• Opens the cached wheel file as a ZIP archive
• Locates the .dist-info directory to get package name and version
• Reads the WHEEL metadata file and uses the first Tag: value when constructing a filename if needed
• Constructs a filename when needed: {name}-{version}-{tag}.whl; falls back to {name}-{version}.whl

Tarball Files

• Opens the cached tarball as a tar archive
• Reads the root directory name which follows the format: {name}-{version}/
• Constructs filename: {name}-{version}.tar.gz

File Sizes

• Uses the .body file size (actual package) instead of metadata file size
• Provides accurate size information for all cached packages

Exclusions

• Files without identifiable package names (HTML index pages, etc.) are automatically excluded
• Only package files (wheels and tarballs) are displayed

Additional notes:

• Operates fully offline; no network requests or server-specific headers
• Compatible with any package index

Testing

Comprehensive test coverage includes:

• Wheel body content extraction with tags
• Tarball body content extraction
• Files without extractable names are excluded
• Body files are skipped correctly
• Corrupted files handled gracefully

All pip cache list tests pass and verify the offline body-content inspection approach.

Feedback & Discussion

All suggestions, reviews, and discussions are welcome. If there are any concerns about naming, option design, or consistency with the existing pip CLI and API, I am happy to refactor or adjust the implementation. The goal is to make this feature both intuitive for end users and maintainable for contributors going forward.

Related Issues

Closes #10460.

This PR directly addresses the problem reported in #10460. If there are other related issues that overlap with this functionality, please feel free to reference them here so they can be resolved by this change as well.

[notatallshaw]

Hi @a-sajjad72, thanks for submitting a PR to pip, please be aware all pip maintainers are currently supporting pip on a volunteer basis and therefore it may be some time before someone can review.

That said I have an early comment:

This PR extends pip cache list to parse HTTP cached responses and extract package information from PyPI's x-pypi-file-* headers, as suggested in the issue discussion.

Pip will not accept a PyPI specific implementation, as it's not a Python packaging standard it won't work on arbitrary indexes and there is no guarantee PyPI will continue to support it in the future.

[a-sajjad72]

Hi @notatallshaw , thanks for the early feedback.

I understand your concern about the current approach being considered “PyPI-specific” because it relies on the x-pypi-file-* headers. My intention wasn’t to hard‑code behavior for PyPI, but I see now that depending on those headers for core functionality effectively ties the feature to PyPI.

I do have an alternative, more index-agnostic idea in mind that would not depend on those headers as the primary source. I suggest we let an initial review happen first (so I know if there are any broader objections), and then we can discuss whether shifting to that alternative approach is the right next step.

If you prefer, I can outline that alternative sooner. Just let me know.

Thanks again for the clarification and your time. Let me know how you would like to proceed.

[notatallshaw]

For myself, I won't be reviewing this PR while it is tied to PyPI specific features, as I would not accept it, and I don't know how much of a change is required to make it index agnostic. Though I won't speak for other maintainers.

[a-sajjad72]

Thanks again. I’ll convert this PR to Draft and refactor it to be index-agnostic before asking for further review.

Planned minimal first step:

• Generic wheel detection (ZIP magic + dist-info/METADATA Name and Version).
• Skip non-artifact / HTML / JSON responses.
• Use any x-pypi-file-* headers only as optional enrichment (never required).
• Placeholder entry if name/version can’t be inferred (or simply skip if you prefer—let me know).
• Keep HTTP listing behind the existing --cache-type flag initially.

If any maintainer would prefer an even smaller scope (e.g. wheels only, no placeholders), please let me know; otherwise I’ll proceed on this basis and update the PR description with a concise design note.

[notatallshaw]

If any maintainer would prefer an even smaller scope

I would advise that the scope be kept as small as possible while still providing a helpful user experience, to be more likely to be accepted. For example, I do not think there should be any use of PyPI only features, even as optional enrichment.

I'm sorry I can't contribute more to a design discussion right now, I don't have much experience here with the design of the cache. Which contributes to why a smaller scope will be easier for a maintainer to start a review.

[pfmoore]

I agree with everything @notatallshaw said. Furthermore, I’d like some discussion of the correctness of the whole approach. The HTTP cache is just that - a cache of HTTP requests, not a cache of downloaded files. The cache includes simple index responses and possibly other information pip has requested - presenting it as just holding wheels is misleading. Also, an index has no obligation to provide any information that a downloaded file comes from a wheel - so we know that accurate data is impossible to achieve, the best we can do is provide a guess. That guess will be accurate in many cases, but we should present it clearly as a guess, and not tempt people to rely on it.

Finally, I’m concerned about the cost of this. Wheels can be big. Have you done any testing of performance, on a large HTTP cache, with some big wheels (multiple copies of PyTorch would be a good start!) in it?

[a-sajjad72]

Thanks @pfmoore for providing your insights on this.

The HTTP cache is just that - a cache of HTTP requests, not a cache of downloaded files. The cache includes simple index responses and possibly other information pip has requested - presenting it as just holding wheels is misleading.

Yeah, I totally agree with you that HTTP caches are just saved HTTP responses and also our required files cached wheels are one of them.

Also, an index has no obligation to provide any information that a downloaded file comes from a wheel - so we know that accurate data is impossible to achieve, the best we can do is provide a guess. That guess will be accurate in many cases, but we should present it clearly as a guess, and not tempt people to rely on it.

When I started working on it, I came to know that some of the cached directories contains responses that are .body and from which many of them are valid archive files.
As I tested, I found total 361 .body responses in the caches from which 134 were valid archive files. And these files include the sdists and bdists collectively.

Finally, I’m concerned about the cost of this. Wheels can be big. Have you done any testing of performance, on a large HTTP cache, with some big wheels (multiple copies of PyTorch would be a good start!) in it?

Yes I tested it, and it (the pypi specific implementation) takes approximately the same time as pip cache info take, maybe a slighter more. but wouldn't take much time.

What will be revised approach?

The core of the revised approach is to identify packages from the .body responses in the HTTP cache, which I've found are often cached wheels (bdist) and source distributions (sdist).

1. More reliable metadata: Instead of parsing the METADATA file, I will extract the package name and version from the normalized .dist-info or .egg-info directory name. This is far more robust as it relies on a consistent, mandatory packaging standard.

2. Support for sdists: The revised approach will ensure that it handles binary distributions (bdists) as well as source distributions (sdists) formats. This provides a more comprehensive view of the cached packages.

3. Performance: I have verified that this method is very efficient, as it avoids extracting the full archive, even for very large files.

4. Unknown files: Any .body responses that are not valid wheel or sdist archives will be ignored, so the output will only contain reliably identified packages.

This approach offers a practical and significantly more reliable way to list cached packages without making incorrect assumptions about the cache's contents.

Please let me know, I will start working on it and update the PR's description.

[a-sajjad72]

@pfmoore The PR description has been updated to reflect the revised HTTP cache listing implementation. Please take a look when you have time, and let me know if anything needs to be changed.