[MAINTENANCE] Add extras to setup.cfg for optional deps #17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tyler-hoffman
force-pushed
the
m/CORE-730/optional-deps
branch
from
4cc27fb to
03302aa
Compare
tyler-hoffman
force-pushed
the
m/CORE-730/optional-deps
branch
from
03302aa to
b92c91d
Compare
tyler-hoffman
force-pushed
the
m/CORE-730/optional-deps
branch
2 times, most recently
from
5bf0cee to
8b68e7c
Compare
tyler-hoffman
force-pushed
the
m/CORE-730/optional-deps
branch
from
8b68e7c to
c139d75
Compare
tyler-hoffman
changed the title
| ruff==0.8.3 | ||
| pytest==8.3.4 | ||
| pytest-mock==3.14.0 | ||
| great-expectations[spark, spark-connect]>=1.3.1 |
There was a problem hiding this comment.
Good question. We do because spark dataframes are in the type signature of at least one of the operators. I don't see a way around it, but LMK if you do.
There was a problem hiding this comment.
Ok...that leads me to ask the question: do we not support pandas for this operator?
There was a problem hiding this comment.
We do support pandas. And that is part of the base gx install, so it's already included. I'll update the description on this PR, since these are good things to be clear about.
NathanFarmer
approved these changes
Jan 14, 2025
|
this looks great, thanks @tyler-hoffman . |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this does
setup.cfgNote on the mypy deps
Our type definitions for dataframes allow for both pandas and spark. The base gx install includes pandas, and the spark extras also need to be included for mypy to work.
What this doesn't do
CORE-730 says we should install the deps and perform a minimal proof that things work. I think the maintenance burden for a reasonable proof is sufficiently high to make this not worth the effort at the current time. We do have proof that the approach works, as the following extras are used by CI:
tests,lint,postgresql,spark.