Overload based on optional argument

[plannigan]

I'm trying to add type hints to a function that is already part of a public API. It is structurally equivalent to:

def thing(value_1, value_2 = None, include_extra=False):
    if value_2 is None:
        value_2 = "empty"
    if include_extra:
        return value_1, value_2, "extra"
    return value_1, value_2

I would like to be able to indicate that the length of returned tuple depends on the value for include_extra. I know that Literal[True] and Literal[False] can be used to help differentiate.
However, I am running into issue because the argument is:

• optional
• not the first optional argument
• not keyword only

At first I tried the following

@overload
def thing(
    value_1: str, value_2: Optional[str] = None, include_extra: Literal[False] = False
) -> Tuple[str, str]:
    ...

@overload
def thing(
    value_1: str, value_2: Optional[str] = None, include_extra: Literal[True] = True
) -> Tuple[str, str, str]:
    ...

but mypy, understandably, says the signatures overlap:

error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]

Also, when I tested it, the type of the return value when include_extra is not given is dependent on which overload is defined first.

Is there a "correct" way to do this? Or is relying on the overload order and silencing the error the best I can do?

[erictraut]

I think what you're looking for is this:

from typing import Literal, overload

@overload
def thing(value_1: str, value_2: str | None = None) -> tuple[str, str]:
    ...

@overload
def thing(
    value_1: str, value_2: str | None, include_extra: Literal[False]
) -> tuple[str, str]:
    ...

@overload
def thing(
    value_1: str, value_2: str | None, include_extra: Literal[True]
) -> tuple[str, str, str]:
    ...

@overload
def thing(
    value_1: str, value_2: str | None, include_extra: bool
) -> tuple[str, str] | tuple[str, str, str]:
    ...

def thing(
    value_1: str, value_2: str | None = None, include_extra: bool = False
) -> tuple[str, str] | tuple[str, str, str]:
    if value_2 is None:
        value_2 = "empty"
    if include_extra:
        return value_1, value_2, "extra"
    return value_1, value_2

[hauntsaninja]

Eric's answer isn't fully correct, since it doesn't permit thing("value", include_extra=True)

Here is something that should be exhaustively correct. It's a little bit annoying, but if you can change your API a little bit you have some opportunities to simplify.

from typing import overload, Literal, Tuple, Optional, Union

# this overload is the straightforward False case
@overload
def thing(value_1: str, value_2: Optional[str] = None, include_extra: Literal[False] = False) -> Tuple[str, str]: ...
# this overload is mostly straightforward, except that we can't include a default value for value_2 because arguments with default values must be at the end
# (if we included a default value for include_extra, the overloads would overlap)
# (if we made include_extra keyword-only, we wouldn't need this overload)
@overload
def thing(value_1: str, value_2: Optional[str], include_extra: Literal[True]) -> Tuple[str, str, str]: ...
# this overload is needed to cover the cases where value_2 is not provided
# (making include_extra keyword-only in this overload is the trick that lets us not give it a default value)
@overload
def thing(value_1: str, value_2: Optional[str] = None, *, include_extra: Literal[True]) -> Tuple[str, str, str]: ...
# this overload is needed to cover the cases where type checkers can't figure out whether include_extra is True or False
# returning a union forces the user to check the return type; in some cases this is an annoying thing and you might want to consider returning Any
@overload
def thing(value_1: str, value_2: Optional[str] = None, *, include_extra: bool) -> Union[Tuple[str, str], Tuple[str, str, str]]: ...

def thing(value_1: str, value_2: Optional[str] = None, include_extra: bool = False):
    if value_2 is None:
        value_2 = "empty"
    if include_extra:
        return value_1, value_2, "extra"
    return value_1, value_2

But yes, definitely annoying. This was my suggestion for a way to fix that wouldn't involve changes to how Python parses code: https://mail.python.org/archives/list/[email protected]/message/3YKFEFTL5GJIEBQLQNIHVEVFWF2FIYNW/

[plannigan]

Thanks for this answer. It is a little frustrating that the answer is "add more overloads", but it makes sense. The actual code was originally written for Python 2.7 and has been evolving since then. So there weren't considerations about "would this API be hard to document statically".

[erictraut]

I get a false positive from pyright here

Pyright is working as designed here. In your example, you omitted the return type from the implementation, so pyright infers the return type from the body. The inferred type doesn't match the return types of the overloads, so it reports the inconsistency. You should provide an explicit return type annotation for the implementation that matches the overloads (Union[Tuple[str, str], Tuple[str, str, str]]).

[gandhis1]

PEP 484 says a non-overloaded definition should be ignored by a type checker:

The @overload-decorated definitions are for the benefit of the type checker only, since they will be overwritten by the non-@overload-decorated definition, while the latter is used at runtime but should be ignored by a type checker.

[igorghi]

I also just hit that issue where pyright is not ignoring the non-overloaded type signatures, which should not be the case according to the official Python docs: https://docs.python.org/3/library/typing.html#typing.overload

@overload-decorated definitions are for the benefit of the type checker only, since they will be overwritten by the non-@overload-decorated definition. The non-@overload-decorated definition, meanwhile, will be used at runtime but should be ignored by a type checker.

[erictraut]

@igorghi, pyright does ignore the implementation signature for purposes of evaluating a call to the overloaded function. However, for purposes of checking consistency between the overloaded signatures and the implementation signature, pyright (necessarily) uses the implementation signature. If you want pyright to skip this consistency check, you can disable it.

[gandhis1]

I know generalities like this aren't particularly useful to anyone, but this proliferation of overloads to exhaustively annotate a function, especially one with default arguments, is an extremely cumbersome requirement. In any non-trivial codebase you will have interfaces like this. One public one where this pattern is common is pandas-stubs. I think there really needs to be a better way, perhaps one that includes some amount of inference by the type checker to fill in missing overloads.

I keep on going back to this issue as an example: python/mypy#5486

In this discussion's example, I'm trying to understand what exactly is ambiguous if you were to permit default arguments across all overloads. Is it only which overload to use if all default arguments are omitted? If so, could we hypothetically could we assume it's the first overload? Also, could the bool overloaded be made unnecessary in some hypothetical change that makes the implementation interface (if annotated) a viable option?

Ultimately, my follow-up question is: is there any context in which we could support only needing two overloads and type checkers do the rest? From a user's perspective the two overloads are what is intuitive and natural, I would argue.

[erictraut]

I think that mypy is being unnecessarily strict in this case.

Here is the example from the cited mypy issue, pasted here for completeness:

from typing import overload
@overload
def func(param: None = ...) -> int: ...
@overload
def func(param: int = ...) -> str: ...

I don't see any potential for misinterpretation or ambiguity in this case because the call expression func() will always match the first overload. Mypy seems to think there is a potential source of ambiguity and generates an error. (By comparison, pyright does not emit an error in this case.)

If the type of param in the first overload were a subtype of param in the second overload, then there would be good reason to report an error because that could lead to an incorrect type evaluation.

Perhaps the mypy maintainers would consider re-opening the issue and relaxing the check so it flags only those cases that are truly ambiguous.