Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] Add extras to setup.cfg for optional deps #17

Merged
merged 9 commits into from
Jan 14, 2025
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make mypy happy again
tyler-hoffman committed Jan 14, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit c139d751cf5f29419b59b416576db9e72e411298
2 changes: 1 addition & 1 deletion .github/workflows/integration-test.yml
Original file line number Diff line number Diff line change
@@ -41,7 +41,7 @@ jobs:
run: python -m pip install --upgrade pip

- name: Install Library
run: pip install ".[tests]"
run: pip install ".[postgresql,tests]"

- name: Setup
run: |
1 change: 1 addition & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
@@ -21,6 +21,7 @@ lint =
mypy==1.14.1
ruff==0.8.3
pytest==8.3.4
pytest-mock==3.14.0
great-expectations[spark, spark-connect]>=1.3.1

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need spark in lint?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. We do because spark dataframes are in the type signature of at least one of the operators. I don't see a way around it, but LMK if you do.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok...that leads me to ask the question: do we not support pandas for this operator?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do support pandas. And that is part of the base gx install, so it's already included. I'll update the description on this PR, since these are good things to be clear about.

gcp =
great-expectations[gcp]>=1.3.1
8 changes: 5 additions & 3 deletions tests/integration/test_validate_dataframe_operator.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
from typing import Callable
from __future__ import annotations

from typing import TYPE_CHECKING, Callable

import pandas as pd
import pytest
from typing import TYPE_CHECKING
from great_expectations import ExpectationSuite
from great_expectations.expectations import ExpectColumnValuesToBeInSet

@@ -13,7 +14,6 @@

if TYPE_CHECKING:
from pyspark.sql import SparkSession
from pyspark.sql.connect.dataframe import DataFrame as SparkConnectDataFrame
from pyspark.sql.connect.session import SparkSession as SparkConnectSession


@@ -90,6 +90,8 @@ def configure_dataframe() -> pyspark.DataFrame:

@pytest.mark.spark_connect_integration
def test_spark_connect(self, spark_connect_session: SparkConnectSession) -> None:
from pyspark.sql.connect.dataframe import DataFrame as SparkConnectDataFrame

column_name = "col_A"
task_id = f"test_spark_{rand_name()}"

1 change: 0 additions & 1 deletion tests/unit/test_validate_batch_operator.py
Original file line number Diff line number Diff line change
@@ -13,7 +13,6 @@
from great_expectations.expectations import (
ExpectColumnValuesToBeInSet,
)
from pytest_mock import MockerFixture

from great_expectations_provider.operators.validate_batch import GXValidateBatchOperator