Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: python typing improvements #2537

Merged
merged 15 commits into from
Sep 9, 2024

Conversation

another-rex
Copy link
Contributor

A series of typing improvements I made while working on the cursor changes. No logic or behavior changes have been made, so should perform identically with the existing deployed code.

Changes:

  • Add mypy-protobuf extension, which enables the --mypy_out= flag for protoc. This generates a typing stub for the grpc protobuf output, which can be used by different python type checking tools/extensions. (does not have to be mypy)
  • Generate and commit the pyi files, and update the readme / script to help generate them in the future.
  • Move helper functions in models.py to their own file (model_helpers.py)
  • Add explicit type definitions to models in models.py
    • These types are not technically "correct", but are the types that will get returned by datastore. This allows us to more easily label the responses from ndb queries.
    • Added an exclusion to the assignment type check of models.py to the pyproject.toml for pyright (used by vscode and pylance). If you use a different type checker, you might want to look into ignoring the errors in models.py in your own config.

Sadly yield still causes problems typing wise, so turning on "strict" typing is still not possible, details for anyone interested in why:

Using the yield anywhere in a function turns the function into a generator,
The left hand side output of yield is received when the caller calls send() on the returned generator.
Python assumes that the sent type will always be the same throughout a single function, so the send type is defined at the top of the function.
ndb uses yield as its own custom async implementation, the left hand side output of yield is the result of the future returned from the right hand side of yield.
This means the output of a yield call will depend on the right hand side of the yield, which is impossible to represent in python's type hinting system.

Copy link
Contributor

@andrewpollock andrewpollock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome code quality improvement, thank you!

osv/model_helpers.py Outdated Show resolved Hide resolved
Copy link
Member

@michaelkedar michaelkedar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, a few very small things

gcp/api/server.py Show resolved Hide resolved
osv/models.py Outdated Show resolved Hide resolved
gcp/api/server.py Outdated Show resolved Hide resolved
@@ -759,7 +765,7 @@ def to_response(b):
else:
context.service_context.abort(grpc.StatusCode.INVALID_ARGUMENT,
'Invalid query.')
return None
raise ValueError
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Either pass a somewhat descriptive string to this ValueError, or, since it should be unreachable, remove it entirely.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment to say that this is unreachable, but is there for the type checker to understand that code does not flow past this point.

osv/models.py Outdated Show resolved Hide resolved
@another-rex another-rex merged commit 5d76ead into google:master Sep 9, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants