Confusing behaviour when using generic type arguments in function

[Dzeri96]

Let's take a look at this function as an example of multiple things not working like I'd assume:

from typing import TypeVar
T = TypeVar('T')

def cast_param_to_type(param_type: type[T], raw_param: str) -> T:
    if param_type in {str, int, float}:
        return param_type(raw_param) # Error: Expected no arguments to "object" constructor
    elif param_type is datetime:
        return datetime.fromisoformat(raw_param) # Error: Type "datetime" is incompatible with type "T@cast_param_to_type"
    else:
        raise ValueError('Unsupported type')

I think the intention behind the implementation is clear: I want to pass a type to the method, and I want it to give me back an instance of it, created with the raw_param as input.

In my opinion, the function, written as is, should be correct, so let's break down the issues that confuse me:

1. If I remove [T] form the param_type typehint, the first error dissapears. How come type and type[T] are handled differently in this regard?
2. Why is the second error even there? If I pass datetime as param_type, this line will return an instance of it. What's the problem?
3. Even if the type checker is being overly cautious, shouldn't the if-statements constrain param_type enough to remove any "doubts" regarding the instanciation?

I'm using Python 3.10 and Pylance v2024.8.2. I know there is a new syntax in 3.12 but I'm building a library and can't lock out so many users at the moment.

[erictraut]

In the first if statement, the type of the expression {str, int, float} evaluates to type set[Unknown] with pyright's default type inference settings. This means the statement param_type in {str, int, float} does not narrow the type of param_type. For information about pyright's set expression inference rules, refer to this documentation. For more details about type narrowing and supported type guard patterns, refer to this documentation.

If you enable strictSetInference (which is enabled by default in "strict" type checking mode), the type of {str, int, float} will be inferred as set[type[str] | type[int] | type[float]], and the x in y type guard pattern should apply, allowing pyright to narrow the type of the expression param_type. Alternatively, you could use a tuple expression here rather than a set expression. This is more idiomatic and will work better with pyright's default tuple inference rules.

Unfortunately, it appears there's a bug in pyright that's causing it to narrow incorrectly in this case. I've logged a bug and will work on a fix for this issue. The reason this works if you change type[T] to type is that the latter is interpreted as type[Any], and the bug appears to apply only in the case where the LHS of the in expression involves a type variable. I'm somewhat surprised no one has reported this issue previously.

In the second elif block, the type of param_type is narrowed to datetime. The expression datetime.fromisoformat(raw_param) evaluates to type datetime, which is not compatible with the return type T. That's because param_type could be a subtype of datetime (that is, it could be an instance of a class that derives from datetime). If you change your code from datetime.fromisoformat to param_type.fromisoformat, you will eliminate this type error.

[Dzeri96]

Thank you very much for explaining the case so clearly. I like it when I dig deep into a topic nobody seems to talk about and it ends up improving the system. Since this is the third time that happened and I've interacted with you, I have to get slightly personal and say that I'm really inspired by your knowledge and constant desire to interact with the community. I don't know how you do it, but there aren't many open source projects with so many resolved issues as pyright. I'm sure I'll have something to report again and you'll answer/fix it in record time!

With that out of the way, I do have a few follow-up questions:

1. Regarding the last point, calling param_type.fromisoformat is semantically the right thing to do in this context, but if it were another case where I wouldn't mind returning datetime even though its child was passed as the type parameter, how would I do it? In java we would use the ? super T syntax.
2. I've ran the "fixed" code through mypy and it doesn't seem to like it. Here are the errors:

:46: error: Too many arguments for "object"  [call-arg]
:48: error: "Type[T]" has no attribute "fromisoformat"  [attr-defined]

I know you also do work on mypy so maybe you'd like to look at these too.

[erictraut]

If you want to return datetime even though its child was passed as the type parameter, you'd need to annotate the return type with datetime rather than T (or a union of the two). You'd probably then need to add an overload to properly represent the fact that datetime is returned only when the input value is a datetime or a subtype therefore.

Yeah, it looks like mypy is also not handling type narrowing in this case. You may want to file a bug report on this. I don't recall seeing this bug reported previously.

Perhaps this pattern (of using in and is operators to look for specific classes) is uncommon. That would explain why no one has reported these issues previously. I think that issubclass is a more typical way to perform this check.

def cast_param_to_type(param_type: type[T], raw_param: str) -> T:
    if issubclass(param_type, (str, int, float)):
        return param_type(raw_param)
    elif issubclass(param_type, datetime):
        return param_type.fromisoformat(raw_param)
    else:
        raise ValueError("Unsupported type")

This type checks fine in pyright, but it runs into a known bug in mypy.

[Dzeri96]

I guess what I was trying to ask was not specific to this example, and was more about setting a lower bound in python (? super T is how you would do it in Java). In any case, it seems like it's not supported right now.

I've created the corresponding bug report with mypy. When/if these are fixed, I'll answer a couple of unanswered stackoverflow questions regarding this.

See you around.