Editable installs not working for meson-python package #323
Comments
|
Hi @peekxc, that looks like missing support in Griffe indeed 🙂 I'll add support for meson, and release it with next Griffe version 🙂 |
|
Hmm, I built your package locally, and I don't get a |
Ahh yes, this was a very recent change---dist was named The dist-name is irrelevent though, I think. The module to import is still (~/primate)$ pip install . --no-build-isolation
(~/primate/docs/src)$ quartodoc build && quarto preview .Builds the docs fine, all tests pass, and all imports get resolved. Meanwhile the editable install does not work: (~/primate)$ pip install --editable . --no-build-isolation
(~/primate/docs/src)$ quartodoc build && quarto preview .
# Fails with quartodoc.builder.utils.WorkaroundKeyError: Cannot find an object named: trace. Does an object with the path primate:trace.hutch exist?
Yes, I get a
Here are the contents: Click to expand `_primate_py_editable_loader.py`# SPDX-FileCopyrightText: 2022 The meson-python developers
#
# SPDX-License-Identifier: MIT
# This file should be standalone! It is copied during the editable hook installation.
from __future__ import annotations
import functools
import importlib.abc
import importlib.machinery
import importlib.util
import json
import os
import pathlib
import subprocess
import sys
import typing
if typing.TYPE_CHECKING:
from collections.abc import Sequence, Set
from types import ModuleType
from typing import Any, Dict, Iterator, List, Optional, Tuple, Union
from typing_extensions import Buffer
NodeBase = Dict[str, Union['Node', str]]
PathStr = Union[str, os.PathLike[str]]
else:
NodeBase = dict
if sys.version_info >= (3, 12):
from importlib.resources.abc import Traversable, TraversableResources
elif sys.version_info >= (3, 9):
from importlib.abc import Traversable, TraversableResources
else:
class Traversable:
pass
class TraversableResources:
pass
MARKER = 'MESONPY_EDITABLE_SKIP'
VERBOSE = 'MESONPY_EDITABLE_VERBOSE'
class MesonpyOrphan(Traversable):
def __init__(self, name: str):
self._name = name
@property
def name(self) -> str:
return self._name
def is_dir(self) -> bool:
return False
def is_file(self) -> bool:
return False
def iterdir(self) -> Iterator[Traversable]:
raise FileNotFoundError()
def open(self, *args, **kwargs): # type: ignore
raise FileNotFoundError()
def joinpath(self, *descendants: PathStr) -> Traversable:
if not descendants:
return self
name = os.fspath(descendants[-1]).split('/')[-1]
return MesonpyOrphan(name)
def __truediv__(self, child: PathStr) -> Traversable:
return self.joinpath(child)
def read_bytes(self) -> bytes:
raise FileNotFoundError()
def read_text(self, encoding: Optional[str] = None) -> str:
raise FileNotFoundError()
class MesonpyTraversable(Traversable):
def __init__(self, name: str, tree: Node):
self._name = name
self._tree = tree
@property
def name(self) -> str:
return self._name
def is_dir(self) -> bool:
return True
def is_file(self) -> bool:
return False
def iterdir(self) -> Iterator[Traversable]:
for name, node in self._tree.items():
yield MesonpyTraversable(name, node) if isinstance(node, dict) else pathlib.Path(node) # type: ignore
def open(self, *args, **kwargs): # type: ignore
raise IsADirectoryError()
@staticmethod
def _flatten(names: Tuple[PathStr, ...]) -> Iterator[str]:
for name in names:
yield from os.fspath(name).split('/')
def joinpath(self, *descendants: PathStr) -> Traversable:
if not descendants:
return self
names = self._flatten(descendants)
name = next(names)
node = self._tree.get(name)
if isinstance(node, dict):
return MesonpyTraversable(name, node).joinpath(*names)
if isinstance(node, str):
return pathlib.Path(node).joinpath(*names)
return MesonpyOrphan(name).joinpath(*names)
def __truediv__(self, child: PathStr) -> Traversable:
return self.joinpath(child)
def read_bytes(self) -> bytes:
raise IsADirectoryError()
def read_text(self, encoding: Optional[str] = None) -> str:
raise IsADirectoryError()
class MesonpyReader(TraversableResources):
def __init__(self, name: str, tree: Node):
self._name = name
self._tree = tree
def files(self) -> Traversable:
return MesonpyTraversable(self._name, self._tree)
class ExtensionFileLoader(importlib.machinery.ExtensionFileLoader):
def __init__(self, name: str, path: str, tree: Node):
super().__init__(name, path)
self._tree = tree
def get_resource_reader(self, name: str) -> TraversableResources:
return MesonpyReader(name, self._tree)
class SourceFileLoader(importlib.machinery.SourceFileLoader):
def __init__(self, name: str, path: str, tree: Node):
super().__init__(name, path)
self._tree = tree
def set_data(self, path: Union[bytes, str], data: Buffer, *, _mode: int = ...) -> None:
# disable saving bytecode
pass
def get_resource_reader(self, name: str) -> TraversableResources:
return MesonpyReader(name, self._tree)
class SourcelessFileLoader(importlib.machinery.SourcelessFileLoader):
def __init__(self, name: str, path: str, tree: Node):
super().__init__(name, path)
self._tree = tree
def get_resource_reader(self, name: str) -> TraversableResources:
return MesonpyReader(name, self._tree)
LOADERS = [
(ExtensionFileLoader, tuple(importlib.machinery.EXTENSION_SUFFIXES)),
(SourceFileLoader, tuple(importlib.machinery.SOURCE_SUFFIXES)),
(SourcelessFileLoader, tuple(importlib.machinery.BYTECODE_SUFFIXES)),
]
def build_module_spec(cls: type, name: str, path: str, tree: Optional[Node]) -> importlib.machinery.ModuleSpec:
loader = cls(name, path, tree)
spec = importlib.machinery.ModuleSpec(name, loader, origin=path)
spec.has_location = True
if loader.is_package(name):
spec.submodule_search_locations = []
return spec
class Node(NodeBase):
"""Tree structure to store a virtual filesystem view."""
def __missing__(self, key: str) -> Node:
value = self[key] = Node()
return value
def __setitem__(self, key: Union[str, Tuple[str, ...]], value: Union[Node, str]) -> None:
node = self
if isinstance(key, tuple):
for k in key[:-1]:
node = typing.cast(Node, node[k])
key = key[-1]
dict.__setitem__(node, key, value)
def __getitem__(self, key: Union[str, Tuple[str, ...]]) -> Union[Node, str]:
node = self
if isinstance(key, tuple):
for k in key[:-1]:
node = typing.cast(Node, node[k])
key = key[-1]
return dict.__getitem__(node, key)
def get(self, key: Union[str, Tuple[str, ...]]) -> Optional[Union[Node, str]]: # type: ignore[override]
node = self
if isinstance(key, tuple):
for k in key[:-1]:
v = dict.get(node, k)
if v is None:
return None
node = typing.cast(Node, v)
key = key[-1]
return dict.get(node, key)
def walk(root: str, path: str = '') -> Iterator[pathlib.Path]:
with os.scandir(os.path.join(root, path)) as entries:
for entry in entries:
if entry.is_dir():
yield from walk(root, os.path.join(path, entry.name))
else:
yield pathlib.Path(path, entry.name)
def collect(install_plan: Dict[str, Dict[str, Any]]) -> Node:
tree = Node()
for key, data in install_plan.items():
for src, target in data.items():
path = pathlib.Path(target['destination'])
if path.parts[0] in {'{py_platlib}', '{py_purelib}'}:
if key == 'install_subdirs' and os.path.isdir(src):
for entry in walk(src):
tree[(*path.parts[1:], *entry.parts)] = os.path.join(src, *entry.parts)
else:
tree[path.parts[1:]] = src
return tree
class MesonpyMetaFinder(importlib.abc.MetaPathFinder):
def __init__(self, names: Set[str], path: str, cmd: List[str], verbose: bool = False):
self._top_level_modules = names
self._build_path = path
self._build_cmd = cmd
self._verbose = verbose
self._loaders: List[Tuple[type, str]] = []
for loader, suffixes in LOADERS:
self._loaders.extend((loader, suffix) for suffix in suffixes)
def __repr__(self) -> str:
return f'{self.__class__.__name__}({self._build_path!r})'
def find_spec(
self,
fullname: str,
path: Optional[Sequence[Union[bytes, str]]] = None,
target: Optional[ModuleType] = None
) -> Optional[importlib.machinery.ModuleSpec]:
if fullname.split('.', maxsplit=1)[0] in self._top_level_modules:
if self._build_path in os.environ.get(MARKER, '').split(os.pathsep):
return None
namespace = False
tree = self.rebuild()
parts = fullname.split('.')
# look for a package
package = tree.get(tuple(parts))
if isinstance(package, Node):
for loader, suffix in self._loaders:
src = package.get('__init__' + suffix)
if isinstance(src, str):
return build_module_spec(loader, fullname, src, package)
else:
namespace = True
# look for a module
for loader, suffix in self._loaders:
src = tree.get((*parts[:-1], parts[-1] + suffix))
if isinstance(src, str):
return build_module_spec(loader, fullname, src, None)
# namespace
if namespace:
spec = importlib.machinery.ModuleSpec(fullname, None)
spec.submodule_search_locations = []
return spec
return None
@functools.lru_cache(maxsize=1)
def rebuild(self) -> Node:
# skip editable wheel lookup during rebuild: during the build
# the module we are rebuilding might be imported causing a
# rebuild loop.
env = os.environ.copy()
env[MARKER] = os.pathsep.join((env.get(MARKER, ''), self._build_path))
if self._verbose or bool(env.get(VERBOSE, '')):
print('+ ' + ' '.join(self._build_cmd))
stdout = None
else:
stdout = subprocess.DEVNULL
subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=stdout, check=True)
install_plan_path = os.path.join(self._build_path, 'meson-info', 'intro-install_plan.json')
with open(install_plan_path, 'r', encoding='utf8') as f:
install_plan = json.load(f)
return collect(install_plan)
def install(names: Set[str], path: str, cmd: List[str], verbose: bool) -> None:
sys.meta_path.insert(0, MesonpyMetaFinder(names, path, cmd, verbose))
install(
{'_random_gen', '_operators', '_orthogonalize', '_trace', 'primate', '_lanczos'},
'/Users/mpiekenbrock/primate/build/cp311',
['/Users/mpiekenbrock/opt/miniconda3/envs/spri/bin/ninja'],
False,
) |
Indeed it's not really relevant for Griffe, but I needed it to know the name pattern of the editable module 🙂 Thanks! So, meson-python's editable modules will now be understood, however there's a second issue: your Python sources have an |
Not sure if I understand entirely the connection to
It definitely does, I can at least confirm I don't know exactly how it works though :) |
Oh, then I'm not sure what's wrong in Quarto, because Griffe was already able to scan the Python sources, given |
|
@peekxc were you able to get it working? Would be interested to hear what was needed! Thanks for debugging / adding the pth support to griffe @pawamoy! With quartodoc, people can use the source_dir option to add sources: quartodoc:
source_dir: "custom_source_directory"It looks like currently quartodoc just adds it to the sys.path. I haven't used it much, so there could be some funkyness there 😅. |
|
As long as sys.path is updated before a Griffe loader is instantiated, this should work :) The loader inits the finder with the given search paths, or sys path if not provided. |
Unfortunately I was not able to get this working; adding a For awhile now I've just been building the docs by building the package normally every time, however I've realized this is just becoming way too slow of a dev loop to maintain. |
|
Ahh, I've figured it out partially. Using the no-package trick from here, I was able to get docs building in an editable install working that auto updated the docs via: quartodoc:
package: null
source_dir: ../src/my_package
sections:
...The issue was that core Python imports for whatever reason interfere with the imports. In my case, I had an The temporary solution I have is to just rename them, though this is suboptimal, as conficts between package namespace happen all the time, and it's probably more pythonic to have these conflicts, e.g. |
I thought I might make this issue for future reference, or for someone else who encounters this, though I admit I don't have either a fix yet.
I'm not able to get
quartodocto load my package once it's in editable mode (on the other hand, it works perfectly fine when not installed in editable mode).In particular,
quartodoc buildfails to find the package modules (I'm working from this package):Related to #12, but also #193 is relevant.
This bug almost surely due to the fact that I'm building with meson python as opposed to setuptools, and the fact that meson_python modifies the
.pthfiles for editable installs (for extension module reasons). In particular, installing e.g. my packageprimatein editable mode puts two files in the virtualenv'ssite_packages:primate_py-editable.pthfile with the contents:_primate_py_editable_loader.pyfile with a bunch of loading-related stuff.I assume quartodoc fails to load the package modules because of 172, but I can't be sure. It's actually not clear if this is a quartodoc issue, a griffe issue, a meson-python problem
The text was updated successfully, but these errors were encountered: