Include prereleases from downstream projects in pip release links pag…

…e. (iree-org#19391)

This fixes iree-org#19193.

Nightly releases of packages from
https://github.com/iree-org/iree-turbine will now be available from
https://iree.dev/pip-release-links.html with (for example)
```bash
pip install --pre --find-links https://iree.dev/pip-release-links.html iree-turbine
```

Previously, this relied on its own index page at
https://github.com/iree-org/iree-turbine/releases/expanded_assets/dev-wheels
. While this link will continue to work, the consolidated page is more
convenient for users.

## Projects included

I'm starting with just iree-turbine. See the notes below about other
packages.

We could still switch to a [PEP 503-compliant package
index](https://peps.python.org/pep-0503/) (usable with
`--extra-index-url`, _not_ `-f, --find-links`), like what PyTorch
[provides](https://pytorch.org/get-started/locally/) at
https://download.pytorch.org/whl/cpu. Note the 50+ packages listed
there, included mirrors of a few versions of all of PyTorch's
dependencies.

## Testing

* Publish action:
https://github.com/ScottTodd/iree/actions/runs/12188757072/job/34002585332
* Pushed to GitHub pages:
https://scotttodd.github.io/iree/pip-release-links.html
* Installed using the release links:

  ```bash
  python3.11 -m venv 3.11.venv
  source 3.11.venv/bin/activate
  
pip install --pre --find-links
https://scotttodd.github.io/iree/pip-release-links.html iree-turbine
--no-deps
# Looking in links:
https://scotttodd.github.io/iree/pip-release-links.html
  # Collecting iree-turbine
# Downloading
https://github.com/iree-org/iree-turbine/releases/download/dev-wheels/iree_turbine-3.1.0rc20241205-py3-none-any.whl
(292 kB)
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.3/292.3 kB 8.1 MB/s eta
0:00:00
  # Installing collected packages: iree-turbine
  # Successfully installed iree-turbine-3.1.0rc20241205
  ```

## Other considerations

We may want to also include downstream packages like
https://github.com/nod-ai/shark-ai (release index page:
https://github.com/nod-ai/shark-ai/releases/expanded_assets/dev-wheels).
I'm currently trying to simplify workflows for that project ([example
here](https://github.com/nod-ai/shark-ai/blob/1763a82a5ebc002d47edb5bc53106f9b312069b9/.github/workflows/ci-shark-ai.yml#L48-L69)).

I left some notes in the workflow file too, but I could see us extending
this to also include torch-mlir and stablehlo packages. The extra
release scraping doesn't cost us much CI time, git storage on the
`gh-pages` branch, or network bandwidth on the website. I also don't
foresee any issues with making it easier to install nightly versions of
those packages. I would pause a bit if the list grows too much beyond
that though.

### Note on the `shark-ai` package

The `shark-ai` meta package (https://pypi.org/project/shark-ai/,
https://github.com/nod-ai/shark-ai/tree/main/shark-ai) pins dependencies
in ways that isn't currently compatible with this package installation
flow. Specifically, the latest nightly releases pin `shortfin==3.1.0`
which can't be resolved by a release candidate like `3.1.0rc20241205`,
so this doesn't work as one might expect:

```bash
pip install --pre --find-links https://iree.dev/pip-release-links.html shark-ai
# Successfully installed shark-ai-3.0.0

# Test with an explicit version to demonstrate the issue:
pip install --pre --find-links https://scotttodd.github.io/iree/pip-release-links.html \
  shark-ai==3.1.0rc20241205
# ERROR: Could not find a version that satisfies the requirement shortfin==3.1.0 (from shark-ai) (from versions: 2.9.0rc20241108, 2.9.0rc20241109, 2.9.0rc20241110, 2.9.0rc20241111, 2.9.0, 2.9.1rc20241112, 2.9.1rc20241113, 2.9.1rc20241114, 2.9.1, 2.9.2rc20241115, 2.9.2rc20241116, 2.9.2rc20241117, 2.9.2rc20241118, 2.9.2, 3.0.0rc20241118, 3.0.0, 3.1.0rc20241119, 3.1.0rc20241120, 3.1.0rc20241121, 3.1.0rc20241122, 3.1.0rc20241123, 3.1.0rc20241124, 3.1.0rc20241125, 3.1.0rc20241126, 3.1.0rc20241127, 3.1.0rc20241128, 3.1.0rc20241129, 3.1.0rc20241130, 3.1.0rc20241201, 3.1.0rc20241202, 3.1.0rc20241203, 3.1.0rc20241204, 3.1.0rc20241205)
# ERROR: No matching distribution found for shortfin==3.1.0
```

.github/workflows/publish_website.yml

@@ -16,7 +16,7 @@ on:
       # Directly authored website source files.
       - "docs/website/**"
       # Python/pip release index page.
-      - "build_tools/scripts/generate_release_index.py"
+      - "build_tools/python_deploy/generate_release_index.py"
       # MLIR dialect definitions and .md generation using iree-tblgen.
       #   Technically this should also include the sources for Tablegen, but
       #   that rarely changes and we want to run this workflow conservatively.
@@ -26,8 +26,18 @@ on:
   release:
     types: [published, unpublished]
 
+  # Run periodically to pick up any documentation changes that were somehow
+  # missed by the above path filters as well as to scrape releases from other
+  # projects that are included on the release index page. Downstream projects
+  # typically build releases in the middle of the night (3AM PST, 11:00 UTC),
+  # so we'll run a bit later than that (6AM PST, 14:00 UTC).
+  schedule:
+    - cron: "0 14 * * *"
+
 jobs:
   publish_website:
+    if: ${{ github.repository_owner == 'iree-org' || github.event_name == 'workflow_dispatch' }}
+
     # Note: a clean build of `iree-tblgen` takes ~5 minutes on standard runners.
     runs-on: ubuntu-20.04
     env:
@@ -53,11 +63,24 @@ jobs:
           pip install requests
           sudo apt update
           sudo apt install -y ninja-build
+
+      # Build a release index page by scraping release package URLs.
+      #
+      # This can scrape the release pages from any public GitHub repositories.
+      # Currently included repositories:
+      #   https://github.com/iree-org/iree
+      #   https://github.com/iree-org/iree-turbine
+      # Ecosystem projects for consideration:
+      #   https://github.com/nod-ai/shark-ai
+      #   https://github.com/openxla/stablehlo
+      #   https://github.com/llvm/torch-mlir (see below)
+      #   https://github.com/llvm/torch-mlir-release
       - name: Generating release index
         run: |
-          ./build_tools/scripts/generate_release_index.py \
-            --repo="${GITHUB_REPOSITORY}" \
+          ./build_tools/python_deploy/generate_release_index.py \
+            --repos="iree-org/iree,iree-org/iree-turbine" \
             --output=docs/website/docs/pip-release-links.html
+
       - name: ccache
         uses: hendrikmuhs/ccache-action@ed74d11c0b343532753ecead8a951bb09bb34bc9 # v1.2.14
         with:

build_tools/python_deploy/README.md

@@ -56,6 +56,23 @@ source .venv/bin/activate
 python -m pip install -r ./pypi_deploy_requirements.txt
 ```
 
+### Release index publication
+
+The [`generate_release_index.py`](./generate_release_index.py) script,
+run as part of
+[`.github/workflows/publish_website.yml`](../../.github/workflows/publish_website.yml),
+scrapes release artifact URLs from https://github.com/iree-org/iree/releases
+(and the release pages for other ecosystem projects) to generate the release
+index published at https://iree.dev/pip-release-links.html.
+
+The release index can be used like so:
+
+```bash
+python -m pip install \
+  --pre --find-links https://iree.dev/pip-release-links.html \
+  iree-base-compiler iree-base-runtime
+```
+
 ## Debugging manylinux builds
 
 We build releases under a manylinux derived docker image. When all goes well,

build_tools/python_deploy/generate_release_index.py

@@ -0,0 +1,145 @@
+#!/usr/bin/env python3
+
+# Copyright 2022 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Creates an HTML page for releases for `pip install --find-links` from GitHub releases.
+
+Sample usage:
+
+    ```bash
+    ./build_tools/python_deploy/generate_release_index.py \
+        --repos=iree-org/iree,iree-org/iree-turbine \
+        --output=docs/website/docs/pip-release-links.html
+    ```
+
+WARNING for developers:
+  GitHub's APIs have rate limits:
+    * 60 requests per hour for unauthenticated users
+    * 5000 requests per hour for authenticated users
+
+  This script only requires read access to public endpoints, but you
+  authenticate yourself to take advantage of the higher rate limit.
+  Creating a fine-grained personal access token with no extra permissions and
+  storing it in the `GITHUB_TOKEN` environment variable seems to work well
+  enough.
+
+  See documentation at:
+    * https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api?apiVersion=2022-11-28#authentication
+    * https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28
+    * https://docs.github.com/en/rest/authentication/authenticating-to-the-rest-api?apiVersion=2022-11-28
+"""
+
+# TODO(#10479) since we're generating this we might as well create a PEP 503
+# compliant index
+
+import argparse
+import html
+import io
+import os
+import requests
+import sys
+import textwrap
+
+GITHUB_TOKEN = os.getenv("GITHUB_TOKEN")
+
+
+def parse_arguments():
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--repos",
+        "--repositories",
+        default="iree-org/iree",
+        help="Comma-delimited list of GitHub repositories to fetch releases from.",
+    )
+    parser.add_argument(
+        "--output",
+        default="-",
+        help="The file to write the HTML to or '-' for stdout (the default)",
+    )
+    return parser.parse_args()
+
+
+class ReleaseFetcher:
+    def __init__(self, repo, per_page=100):
+        self._session = requests.Session()
+        self._repo = repo
+        self._per_page = per_page
+
+        self.headers = {
+            "Accept": "application/vnd.github+json",
+            "X-GitHub-Api-Version": "2022-11-28",
+        }
+        if GITHUB_TOKEN:
+            self.headers["Authorization"] = f"Bearer {GITHUB_TOKEN}"
+
+    def get_all(self):
+        url = f"https://api.github.com/repos/{self._repo}/releases"
+        page = 1
+
+        # GitHub limits API responses to the first 1000 results.
+        while page * self._per_page < 1000:
+            response = self._session.get(
+                url,
+                params={
+                    "page": page,
+                    "per_page": self._per_page,
+                },
+                headers=self.headers,
+            )
+            if response.status_code != 200:
+                raise RuntimeError(
+                    f"Request was not successful, reason: {response.reason}"
+                )
+            for release in response.json():
+                yield release
+            if "next" not in response.links:
+                break
+            page += 1
+
+
+def add_releases_for_repository(repo: str, file: io.TextIOWrapper):
+    fetcher = ReleaseFetcher(repo)
+
+    file.write(
+        f'    <h2>Packages for <a href="https://github.com/{repo}">{repo}</a></h2>\n'
+    )
+
+    for release in fetcher.get_all():
+        if release["draft"]:
+            continue
+        for asset in release["assets"]:
+            url = html.escape(asset["browser_download_url"])
+            name = html.escape(asset["name"])
+            file.write(f"    <a href={url}>{name}</a><br>\n")
+
+
+def main(args):
+    repos = args.repos.split(",")
+    with sys.stdout if args.output == "-" else open(args.output, "w") as f:
+        f.write(
+            textwrap.dedent(
+                """\
+            <!DOCTYPE html>
+            <html>
+              <body>
+            """
+            )
+        )
+        for repo in repos:
+            add_releases_for_repository(repo, f)
+        f.write(
+            textwrap.dedent(
+                """\
+      </body>
+    </html>
+    """
+            )
+        )
+
+
+if __name__ == "__main__":
+    main(parse_arguments())

docs/website/.gitignore

@@ -0,0 +1,2 @@
+# Produced by build_tools/python_deploy/generate_release_index.py
+docs/pip-release-links.html