Reveal metaclass base-class to type-checker

[scanny]

I'm wondering if there's a way to let the type-checker know that all classes produced by a certain metaclass have a specified base-class.

I use a metaclass to create lxml custom-element-classes.

• Each element-class proxies an XML element (an lxml.etree.ElementBase instance).
• I include lxml.etree.ElementBase as a base in the .__new__() method of the metaclass.
• The element-classes work fine and have for a long time.

The problem is that pyright (and I expect mypy too) doesn't recognize the ElementBase base-class in the MRO of the element-classes produced. So when I use an ElementBase method on the element-class it doesn't type-check, producing
an UnknownMemberType diagnostic.

I'm suspecting that I need to annotate the .__new__() method of the metaclass differently. Here's what it looks like now:

_T = TypeVar("_T")

class MetaOxmlElement(type):
    """Metaclass for BaseOxmlElement."""

    def __new__(
        cls: Type[_T], clsname: str, bases: Tuple[type, ...], namespace: Dict[str, Any]
    ) -> _T:
        bases = (*bases, etree.ElementBase)
        return super().__new__(cls, clsname, bases, namespace)

More Context

I'm the author of python-docx. Because the .docx format is XML-based and uses hundreds of distinct XML element types, I use a declaratively-defined class for each XML element class (these become lxml "custom-element-classes"). This is a typical one: https://github.com/python-openxml/python-docx/blob/8e05650be5da0820a6591698e675164fe70d50a8/src/docx/oxml/text/parfmt.py#L28

The metaclass that creates these element-classes is MetaOxmlElement here:
https://github.com/python-openxml/python-docx/blob/8e05650be5da0820a6591698e675164fe70d50a8/src/docx/oxml/xmlchemy.py#L91-L112

[erictraut]

I don't think there's a way in the current type system to handle such dynamic code.

Why do you dynamically inject the ElementBase base class in an override to the metaclass __new__ method? Could you instead statically specify ElementBase as a base class for BaseOxmlElement? That is, delete the MetaOxmlElement.__new__ method and then change the BaseOxmlElement class definition to the following:

class BaseOxmlElement(ElementBase, metaclass=MetaOxmlElement):
    ...

I think that achieves the same thing without needing to resort to dynamic "magic" that confuses a static type checker.

[scanny]

Thanks very much Eric for replying :)

Making that change gives me: The metaclass of a derived class must be a subclass of the metaclasses of all its base classes [generalTypeIssue].

I've seen that before while trying one thing and another to see if I could happen on a magic combination. I can't say I actually know what it means though.

The MRO for lxml.etree.ElementBase is:

>>> from lxml import etree
>>> etree.ElementBase.__mro__
(<class 'lxml.etree.ElementBase'>, <class 'lxml.etree._Element'>, <class 'object'>)
>>> etree._Element.__mro__
(<class 'lxml.etree._Element'>, <class 'object'>)

I get that same typing error with both ElementBase and _Element, but not with object.

Neither ElementBase or _Element is a metaclass as far as I can tell, but _Element appears to be a .pyx C-binding thing. I'm not really familiar with those but I believe this is its definition: https://github.com/lxml/lxml/blob/master/src/lxml/etree.pyx#L703

ElementBase appears to be defined here: https://github.com/lxml/lxml/blob/master/src/lxml/classlookup.pxi#L6

[scanny]

@erictraut Yes, this is actually exactly what I have after removing the __new__() method from MetaOxmlElement.

This produces the "... derived class must be a subclass ..." error on the class BaseOxmlElement(...) line. Are you saying I should just ignore that one type-error and it should be fine?

It does have the desired effect of now recognizing the etree.ElementBase base class in all the element-classes, which is a big win.

And it still passes all the (very thorough) unit and acceptance tests, so I'd have to say it is fine as far as I can tell :)

So looking to understand this a little better, are you attributing that type-error to:

• ElementBase has a metaclass (because all classes have a metaclass), assumedly type, unless C-binding code is somehow different in that regard.
• BaseOxmlElement has a metaclass of MetaOxmlElement.
• MetaOxmlElement is not a subclass of type.

Okay, so based on the contradiction there (MetaOxmlElement IS a subclass of type), the initial assumption must be mistaken, meaning that ElementBase either does not have a metaclass or it is something other than type.

Just doing some quick research, it looks like C-extension functions at least don't automatically have type as their metaclass and may not have one at all, something to do with PyTypeObject not having a .tp_meta field or something.

Okay, so here's my conclusion.

• C-extension classes, such as lxml.etree.ElementBase will at least sometimes not have type as their metaclass.
• For this reason, using such a C-extension class as a base-class where the subclass also declares a metaclass will trigger a "technically correct" but probably not meaningful ".. conflicting metaclasses .." type error.
• Ignoring that error is probably fine as long as it works.

Does that sound about right to you @erictraut?

[erictraut]

I mistakenly understood that you were receiving this error at runtime, but it sounds like you are seeing this as a type error coming from pyright. Is that correct?

I'm not able to repro that error with pyright. Are you able to repro it with my second code sample above? I also tried to repro it in the python.docx repo by manually applying my suggested change, but I'm not seeing that error. Here are some follow-up questions:

• Do you have type stubs for the lxml library? This is an untyped library, and it's mostly implemented in C, so there's no type information available. Perhaps you've installed the lxml-stubs stub library? I tried to repro both with and without these stubs.
• Where does this error appear within the code? Is it when you attempt to define a subclass of BaseOxmlElement?
• Can you create a minimal, self-contained sample that repros the problem you're seeing?
• Are you using a recent version of pyright? If not, please make sure to upgrade.

[scanny]

H Eric, I was using v1.1.308 but I still get that type-error after updating to v1.1.331. I get it both in my editor (vim) and at the command-line.

I'm on an Intel Mac running Big Sur 11.6.1.

lxml install is this, latest of both:

:) ± pip freeze | grep lxml
lxml==4.9.3
lxml-stubs==0.4.0

Minimal reproducible code is this:

from lxml.etree import ElementBase


class MetaOxmlElement(type):
    pass


class BaseOxmlElement(ElementBase, metaclass=MetaOxmlElement): 
# ^^^scratch.py:8:7 - error: The metaclass of a derived class must be a subclass of the metaclasses of all its base classes (reportGeneralTypeIssues)

    pass

pyrightconfig.json is this:

{
    "exclude": [
        "**/__pycache__",
        "**/.*"
    ],
    "ignore":  [
    ],
    "include": [
        "src/docx/",
        "tests"
    ],
    "pythonPlatform": "All",
    "pythonVersion": "3.7",
    "reportImportCycles": true,
    "reportUnnecessaryCast": true,
    "reportUnnecessaryTypeIgnoreComment": true,
    "stubPath": "./typings",
    "typeCheckingMode": "strict",
    "useLibraryCodeForTypes": true,
    "verboseOutput": true
}

I tried turning useLibraryCodeForTypes false but no change. I've been flipping that back and forth lately as I work on adding the typing, but generally keep it off.

I always type strict. That's where the real value is as far as I can tell.

./typings contains some minimal type stubs for the behave package I use for "Cucumber" acceptance tests.

Let me know if there are any more details you need :)

[erictraut]

Thanks, I'm able to repro the issue now. The problem is that the lxml stubs indicate that lxml.etree._Element subclasses from Sized, which uses an ABCMeta metaclass. At runtime, lxml.etree._Element does not derive from Sized, so this appears to be a bug in the lxml type stubs, which are not accurately reflecting the implementation. You can work around the issue in your code by adding a # type: ignore or # pyright: ignore.

It occurs to me that pyright's error message in this case could be improved. I've updated the code accordingly, and future releases of pyright will now report the specific metaclasses that are causing the conflict.

[scanny]

This is terrific! Thanks so much Eric!! :)