feat: Scanning plugin entrypoints of external packages

[jopemachine]

Fix #2359, refs #1683.

Wheel distributions

This PR implements an automated process to extract .whl files found under the {base-dir}/wheelhouse directory and add their source directories to PYTHONPATH, so that they could be loaded without additional installation steps.

Example:

• Download a wheel package into ./wheelhouse directory.
• Upon scanning, it will be extracted as sub-directories of ./wheelhouse.
• The package appears in the scan result from the "python-package" source.

Tip

To distribute and install plugins that have external dependencies that the core does not use, you may utilize the pip wheel command to create a collection of wheel files and put it under the wheelhouse directory, as the plugin loader will load all of them into PYTHONPATH.

Note

When a Python wheel file is extracted, it contains:

• The top-most package directory (For our packages, it is usually ai.)
• {pkg-name}-{version}.dist-info/
  • entry_points.txt
  • METADATA
  • RECORDS
  • top_level.txt
  • WHEEL
• {pkg-name}.libs/ (empty)

We may simply insert the extracted directory into sys.path to make it importable, assuming that all dependencies of the wheel is already available in the venv.

Plugin source checkouts

It also enhances the backend.ai plugin scan command (#2070) to explicitly display the plugin-checkout sources while skipping plugin-checkout sources when actually loading the plugins. Also added more detailed guides such as [agent].allowed-compute-plugins checks when the group name contains "accelerator".

You may enable the debug logging of the scan command itself by adding --debug option to the plugin subcommand, like ./backend.ai plugin --debug scan ....

Example:

• Clone a private cuda plugin into the ./plugins directory.
• Run ./backend.ai plugin scan backendai_accelerator_v21:

• Run ./py -m pip install -e plugins/cuda.
• Run ./backend.ai plugin scan backendai_accelerator_v21:

---

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Mention to the original issue

---

📚 Documentation preview 📚: https://sorna--2377.org.readthedocs.build/en/2377/

---

📚 Documentation preview 📚: https://sorna-ko--2377.org.readthedocs.build/ko/2377/

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “flow:merge-queue” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “flow:hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[jopemachine]

• feat: Scanning plugin entrypoints of external packages #2377 👈
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @jopemachine and the rest of your teammates on Graphite

[achimnol]

Um.. we need to support the scenario for developing plugins by cloning the repositories into plugins/ directory.
Do you have any better idea to it?

[jopemachine]

Um.. we need to support the scenario for developing plugins by cloning the repositories into plugins/ directory. Do you have any better idea to it?

Even if scan_entrypoint_from_plugin_checkouts is removed from the development environment, each plugin's path under the plugins directory is added to PYTHONPATH from ./py, so it operates the same as before.

That is, all plugins under the plugins directory are still recognized as plugins, and we can install and develop plugins in an editable way through the ./scripts/install-plugin.sh script in the development environment.

[achimnol]

I'm also considering to fix scie-based executables suffer from missing imports when executed inside the working copy, while they work fine when executed outside.

[achimnol]

Um.. we need to support the scenario for developing plugins by cloning the repositories into plugins/ directory. Do you have any better idea to it?

Even if scan_entrypoint_from_plugin_checkouts is removed from the development environment, each plugin's path under the plugins directory is added to PYTHONPATH from ./py, so it operates the same as before.

That is, all plugins under the plugins directory are still recognized as plugins, and we can install and develop plugins in an editable way through the ./scripts/install-plugin.sh script in the development environment.

This is not true.

Expected result:

[achimnol]

Okay, I'm going to preserve the plugin-checkout type for debugging purpose (the CLI), while skipping it in the loader mechanism.