Method overrides class in an incompatible manner.

[mrendi29]

Hi All,

I have started exploring pyright as a static type checker recently and have had a pleasant experience.
I am running into some issues which I would need some help with.

I am currently specifying an abstract class like this:

from abc import ABC, abstractmethod
from typing import Generic, TypeVar

from pandas import DataFrame as pdf
from pyarrow import RecordBatch, Table
from pyspark.sql import DataFrame as sdf

# Data can be one of each through the ETL steps
DataType = TypeVar("DataType", RecordBatch, Table, pdf, sdf) 

# Option 1
class Pipeline(ABC):
    """Base Pipeline Structure Abstract Class"""

    @abstractmethod
    def extract(self) -> DataType:
        pass

    @abstractmethod
    def transform(self, data: DataType) -> DataType:
        pass

    @abstractmethod
    def load(self, data: DataType) -> bool:
        pass

# Option 2
class GenericPipeline(ABC, Generic[DataType]):
    """Base Pipeline Structure Abstract Class"""

    @abstractmethod
    def extract(self) -> DataType:
        pass

    @abstractmethod
    def transform(self, data: DataType) -> DataType:
        pass

    @abstractmethod
    def load(self, data: DataType) -> bool:
        pass


class ConcretePipeline(Pipeline):
    """
    Method "load" overrides class "Pipeline" in an incompatible manner
    Parameter 2 type mismatch: base parameter is type "DataType@load", override parameter is type "RecordBatch"
    Type "RecordBatch* | Table* | DataFrame*" is incompatible with type "RecordBatch"
    "DataFrame*" is incompatible with "RecordBatch"PylancereportIncompatibleMethodOverride
    """

    def load(self, data: RecordBatch) -> bool:
        return True


class ConcretePipelineOption2(GenericPipeline[Table]):
    """
    Method "load" overrides class "GenericPipeline" in an incompatible manner
    Parameter 2 type mismatch: base parameter is type "Table", override parameter is type "RecordBatch"
    "Table" is incompatible with "RecordBatch"PylancereportIncompatibleMethodOverride
    """

    def load(self, data: RecordBatch) -> bool:
        return True

What i am basically trying to achieve is having a BasePipeline class that specifies the structure of each pipeline (subclass) in the ETL process. The reason why i create a DataType TypeVar is because the data can go through various transformations in each step of the process.
I could start with a pyarrow.Table in extract, have a pandas dataframe in transform and end up with pyarrow.RecordBatch in load.
Using a generic class does not help either as that limits me to the generic type i am using.

Is there anyway to make such a think work with pyright?

Looking forward to your ideas.

[erictraut]

I'm glad that pyright is working well for you.

Pyright is a standards-compliant type checker in that it endeavors to comply with the rules set forth in the official Python typing spec.

If you have general questions about the static type system or would like guidance about how to model your code using static types, the Python typing forum is a good place to post your questions because they'll get broad visibility from members of the broader Python typing community.

That said, I'll do my best to answer your question here.

First, I'm assuming that you have the pyarrow-stubs stub library installed. Without this, I wasn't able to repro the errors you're seeing.

I noticed that you're trying to use a value-constrained TypeVar. These can be useful in rare circumstances, but they are typically not the right way to model your types. It's definitely not a tool I would reach for as a first choice. A simple union type or a regular TypeVar that has an upper bound of a union type is probably a better choice here, depending on what you're trying to achieve.

It's not clear to me from your description whether you intend to instantiate new Pipeline objects for each step of the pipeline. I'm guessing the answer is no. If my assumption is correct, then it doesn't make sense to parameterize Pipeline with a particular data type, since the data type will change over the course of the pipeline. If that's a correct assumption, then a simple union type is the right tool here. There's no need in that case to use any TypeVar or make your classes generic.

type DataType = RecordBatch | Table | pdf | sdf

class Pipeline(ABC):
    @abstractmethod
    def extract(self) -> DataType:
        pass

    @abstractmethod
    def transform(self, data: DataType) -> DataType:
        pass

    @abstractmethod
    def load(self, data: DataType) -> bool:
        pass

class ConcretePipeline(Pipeline):
    def load(self, data: DataType) -> bool:
        return True

If my assumptions are incorrect or I misunderstood the constraints of your problem, let me know and I'll take another swing at answering your question.

[mrendi29]

Hi Eric, Thank you for the detailed answer. I really appreciate it! Your understanding of my usage is correct, I only instantiate the Pipeline once and i call a run() method which calls extract->transform->load one by one.
The solution that you have proposed is very elegant and indeed works.

Out of curiosity, why is it not possible in this case to not use a concrete type in the function argument directly? That would in theory tell the developer what type of data to exactly expect as well:

class ConcretePipeline(Pipeline):
    def load(self, data: pandas.Dataframe) -> bool:
        return True

Is this such a thing possible or is this against a rule of python's type system?

[erictraut]

is this against a rule of python's type system?

If you define a method in a base class and then override that method in a subclass, the override method must be a subtype of the overridden method. It must accept all legal arguments accepted by the base class. If you think about it, that makes sense because any class that purports to be compatible with the base class must accept all of the same arguments as the base class method. If the base class method indicates that it accepts an argument of type RecordBatch | Table | pdf | sdf but the child method accepts only pdf, then your program could crash if someone passes it a RecordBatch, Table or an sdf. For details about subtyping rules for callables, refer to this section of the Python typing spec.