Equivalent of mypy "allow_redefinition"?

[ronf]

I'm trying to convert from using "mypy" to "pyright" to type-check a project (to be able to use ParamSpec), and I'm running into some snags making "pyright" happy with the code even after getting "mypy" to run cleanly.

In particular, is there an equivalent to mypy's "allow_redefinition" in pyright? There are times when I couldn't get "mypy" to figure out that the type of a variable had been narrowed in the code, and so I would add explicit redefinitions to the narrower type when needed to give the type checker some help. However, I don't see an option like that in pyright, and when I try I get back an error about the parameter declaration obscuring a previous declaration of the same name.

Also, there are places where mypy seems to be better at figuring things out on its own. For instance, I had a block which looked something like:

from typing import Optional, Union

_DEFAULT_PATH = 'abc'

class FooTransport:
    pass

class BarTransport:
    def __init__(self, path: str):
        self._path = path

def foo(path: Optional[str]):
    transport: Union[None, FooTransport, BarTransport] = None

    if not path:
        try:
            transport = FooTransport()
        except OSError:
            path = _DEFAULT_PATH

    if not transport:
        transport = BarTransport(path)

With mypy, it was able to figure out that path became a str (rather than an Optional[str]) in the second "if", since the first "if" either assigned transport to be non-null or it assigned path to be non-null. With pyright, though, I got back the error:

  /private/tmp/test.py:24:34 - error: Argument of type "str | None" cannot be assigned to parameter "path" of type "str" in function "__init__"

Without being able to manually redefine the type of path, how can I get around this error?

[erictraut]

I don't think it's safe for a type checker to assume that path is narrowed to a str after the first if block. If no OSError exception is raised, path will retain its value of None.

If you are sure that your logic is sound and there's no way for path to retain its None value at the end of the first if statement, then I recommend adding an assert statement to verify this assumption. This will also tell pyright about your assumption and allow it to apply the type narrowing.

    if not path:
        try:
            transport = FooTransport()
        except OSError:
            path = _DEFAULT_PATH
        
        assert path is not None

[ronf]

Thanks very much for your response, Eric!

When I first wrote this code and ran "mypy" on it, I actually had an explicit path: str redefinition between the two "if" statements. So, I was wrong about mypy figuring that out on its own. It turned out that I later added strict_optional = False to the mypy config file, and that ended up eliminating the need for the narrowing in that specific case.

You're right that the first "if" block alone is not enough to guarantee agent_path is not None. It was only the combination of the first block plus the check that transport was still not set in the second if that made it safe, but that's a lot to ask the type checker to figure out.

The "assert" looks like a good option in this case -- I didn't realize that would be enough to signal the type checker. It seems to work well in the quick test that I did in both mypy and pyright.

After addressing other issues, I did plan to go back and turn on strict_optional again on mypy, but it's going to take some work. Doing so generates 500+ mypy errors in my current code base, and it looks like pyright also reports these errors. Do you know if there is a pyright equivalent for mypy's strict_optional = False I can use until I can address those issues?

[erictraut]

Pyright has several configuration options that control the way it deals with Optional types.

You can enable or disable each of these independently in the pyrightconfig.json or pyproject.tolm file. For full documentation, check out this link. I've pasted the relevant configuration items below for convenience.

reportOptionalSubscript [boolean or string, optional]: Generate or suppress diagnostics for an attempt to subscript (index) a variable with an Optional type. The default value for this setting is 'error'.

reportOptionalMemberAccess [boolean or string, optional]: Generate or suppress diagnostics for an attempt to access a member of a variable with an Optional type. The default value for this setting is 'error'.

reportOptionalCall [boolean or string, optional]: Generate or suppress diagnostics for an attempt to call a variable with an Optional type. The default value for this setting is 'error'.

reportOptionalIterable [boolean or string, optional]: Generate or suppress diagnostics for an attempt to use an Optional type as an iterable value (e.g. within a for statement). The default value for this setting is 'error'.

reportOptionalContextManager [boolean or string, optional]: Generate or suppress diagnostics for an attempt to use an Optional type as a context manager (as a parameter to a with statement). The default value for this setting is 'error'.

reportOptionalOperand [boolean or string, optional]: Generate or suppress diagnostics for an attempt to use an Optional type as an operand to a binary or unary operator (like '+', '==', 'or', 'not'). The default value for this setting is 'error'.

[ronf]

I found those, but I'm not sure they'll do what I need here. In particular, the mypy option allows a type to be not declared as Optional, but still allow None to be assigned to it (basically making None an allowed value for all types). I do assignments to None today in places where I close something like a socket or file and want to make it clear the target is closed. That's a case where the variable probably should be explicitly Optional and the code should be prepared to deal with the possibility of it being None, but as I said it may take me a while to fix up all such places in the code, as adding the Optional wrapper around the type then ends up causing complaints in other places which weren't expecting an optional type.

The other use case here where an __init__ method does an initial assignment of instance variables to None if those values aren't set until later in some other method of the class. Once set to a proper value, the member should remain set for the lifetime of the object. This is a case where it would actually be nice not to have to make the type Optional, though I understand there is theoretically a window in where the member could end up being accessed while it is still set to None.

[erictraut]

I think you'll find that the assert statement will work well for these cases. It will not only inform the type checker of your assumptions, but it will also perform runtime checks of your assumptions when you run the code in debug mode. Assert statements also inform other maintainers of your code what assumptions you were making — an enforceable comment of sorts. In my experience asserts are easy to add to a code base, and it doesn't take that many of them to eliminate a lot of type errors. Eliminating 500 errors within a code base shouldn't take more than an hour or two, and your code base will be much more robust once it's done. I recommend using pyright in interactive mode within VS Code or another smart editor so you get instantaneous feedback when errors are fixed.

[ronf]

Thanks again, Eric - I really appreciate your help here. I agree that this is work that will definitely pay off going forward. So far, a lot of the places I've found actually were already expecting and handling None properly, so all I needed was to add an explicit Optional without even needing an assert. I'll definitely look at using the assert when dealing with the use cases I mentioned above, though, or in places where I want to narrow the type after handling the None in another code path.