How to use pandera to validate non numeric columns against numeric datatype in schema and raise error

[soumen-ghosh]

I'm trying pandera for the 1st time. How can I use pandera to validate dataframe columns to ensure errors are raised if data-type does not match against the schema?

Say, I have the schema defined as following -

import re
from typing import Any, Dict, List, Pattern

# import jsonschema
import pandas as pd
import numpy as np
import pandera as pa

pa_schema_dict: Dict[str, pa.Column] = {
    item[0]: pa.Column(
        dtype=item[1],
        nullable=True,
        required=True,
        name=item[0],
        drop_invalid_rows=True,
    )
    for item in [("housing_median_age", "int"), ("ocean_proximity", "int")]
}

for name, value in pa_schema_dict.items():
    if value.dtype.type in ("float", "int"):
        print(f"{name}: {value.dtype.type}")
        # check: pa.Check = pa.Check(
        #     lambda x: (
        #         True
        #         if re.match(
        #             pattern=r"^-?\d+(\.\d+)?([eE]-?\d+)?$",
        #             string=str(x),
        #             flags=re.ASCII,
        #         )
        #         is not None
        #         else False
        #     ),
        # )
        check: pa.Check = pa.Check(
            pd.api.types.is_numeric_dtype,
        )
        if value.checks:
            value.checks.append(check)
        else:
            value.checks = [check]

pa_feature_schema: pa.DataFrameSchema = pa.DataFrameSchema(
    columns=pa_schema_dict,
    name="data_schema",
    drop_invalid_rows=True,
    coerce=True,
    strict="filter",
    add_missing_columns=False,
)

And say, my dataframe is like following:

df: pd.DataFrame = pd.DataFrame(
    {"housing_median_age": [6, 25, np.nan, 25],
     "ocean_proximity": ["NEAR BAY", "INLAND", "<1H OCEAN", "NEAR OCEAN"]
     })

Since "ocean_proximity" in schema is of type int64 but in the dataframe it is object, I want an error to be raised.

However, when I try the below code, it does not raise any exception

try:
    validated_df: pd.DataFrame = pa_feature_schema.validate(
        check_obj=df, lazy=True, inplace=True
    )
except (pa.errors.SchemaError, pa.errors.SchemaDefinitionError,) as e:
    print(str(e))

How to achieve this?

Since the dataframe could be any random dataframe, I get the schema through a user config file. I'm not sure how to use DataFrameModel to define schema programmatically in such case.

[soumen-ghosh]

I should NOT say this is the answer, but I am, for now, using the below work around by calling this method right after the pa_feature_schema.validate() call. However, I'll look forward to proper solution.

   def validate_dtypes(
        df_: pd.DataFrame, schema_cols_: Iterable[pa.Column]
    ) -> NoReturn:

        invalid_cols: List[str] = []
        for col in schema_cols_:
            if np.issubdtype(col.dtype.type, np.number):
                if not pd.api.types.is_numeric_dtype(df_[col.name]):
                    invalid_cols.append(col.name)
            else:
                # TODO: validate for other types # pylint: disable=W0511
                pass

        if invalid_cols:
            raise TypeError(
                f"Dataframe columns '{invalid_cols}' are not of numeric type "
                f"but defined as numeric type in the schema"
            )

PS: I have used TypeError as I am not sure what to pass to data in the constructor of pa.errors.SchemaError