v0.14.0: Pandera Internals Rewrite [phase 1]

[cosmicBboy]

⭐️ Highlights

The main highlight of this release is that phase 1 of the Pandera internals re-write is complete 🎉🚀! This is a backwards-compatible re-write (unit tests FTW 😅) that should just work with your existing pandera code. Please submit bug reports if you encounter any regressions that weren't covered by the current test suite.

These PRs #913 #1109, and #1110 address #381, and essentially decouples pandas-specific logic from the pandera schema specification. In summary:

• The pandera schema specifications are defined in pandera.api, containing:
  • schema base classes in pandera.api.base
  • pandera schema classes in pandera.api.pandas
  • the global check and hypothesis namespace in pandera.api.checks.Check and pandera.api.hypotheses.Hypothesis
  • decorators are provided in pandera.api.extensions to be able to register builtin and custom checks/hypotheses
• The pandera backend validation logic is defined in pandera.backends, containing:
  • backend base classes in pandera.backends.base
  • pandas-specific backend validators in pandera.backends.pandas

Now, all pandas-specific logic is isolated to specific modules, where support for additional non-pandas-compliant schema specifications and their associated backends can be implemented either as 1st-party-maintained libraries (see issues for supporting polars and ibis) or 3rd party libraries.

🛣 Rewrite Roadmap

The bulk of the re-write is complete, however there are still some outstanding items:

• Write validation backends for the existing pandas-like frameworks (dask, pyspark.pandas, modin). This may lead to refactoring some of the abstractions that came out of the rewrite.
• Write an alpha version of the pandera-ibis package, which will create a schema specification and validation backends for ibis data structures (see issue Support Ibis Backend #1105)
• Document the process of writing your own 3rd party libraries based on pandera for any arbitrary statistical data container.

What's Changed

• Bugfix/996: strict="filter" doesn't work on spark dataframes by @nwoodbury in Bugfix/996: strict="filter" doesn't work on spark dataframes #1001
• unpin pandas-stubs version by @williamjamir in unpin pandas-stubs version #1000
• add PR messages, DCO to contributing guide by @cosmicBboy in add PR messages, DCO to contributing guide #1006
• Turn failure-cases to string to avoid hashing unhashable objects by @a-recknagel in Turn failure-cases to string to avoid hashing unhashable objects #1014
• not require coerce==True when for PydandticModels by @the-matt-morris in not require coerce==True when for PydandticModels #1011
• Schema Model manipulation docs by @a-recknagel in Schema Model manipulation docs #1012
• Fix handling of decimals with scale=0 by @a-recknagel in Fix handling of decimals with scale=0 #1010
• Add Union support to check_types: Bugfix/977 by @kr-hansen in Add Union support to check_types: Bugfix/977 #995
• Bugfix/997 by @joepatol in Bugfix/997 #1017
• update mypy plugin and tests by @cosmicBboy in update mypy plugin and tests #1007
• fix issue where @check_types-decorated function is an iterable by @cosmicBboy in fix issue where @check_types-decorated function is an iterable #1022
• fix mypy extra unit tests, pin pandas-stubs for dev env by @cosmicBboy in fix mypy extra unit tests, pin pandas-stubs for dev env #1056
• Feature/511: Copy columns in DataFrameSchema.init() by @NickCrews in Feature/511: Copy columns in DataFrameSchema.__init__() #1055
• Unpinning ray from requirements-dev.txt by @erichamers in Unpinning ray from requirements-dev.txt #1052
• core and backend pandera API internals rewrite by @cosmicBboy in core and backend pandera API internals rewrite #913
• Small fix to example by @brl0 in Small fix to example #1083
• Coerce dt indexes and series by @cristianmatache in Coerce dt indexes and series #1057
• correctly type-check strings by @cosmicBboy in correctly type-check strings #1106
• fix lazy validation issue with regex columns if no column found by @cosmicBboy in fix lazy validation issue with regex columns if no column found #1107
• fix(dtypes.py): correction at function is_numeric docstring by @HenriqueAJNB in fix(dtypes.py): correction at function is_numeric docstring #1100
• internals rewrite: clean up checks and hypothesis functionality by @cosmicBboy in internals rewrite: clean up checks and hypothesis functionality #1109
• rename pandera.core to pandera.api by @cosmicBboy in rename pandera.core to pandera.api #1110

New Contributors

• @nwoodbury made their first contribution in Bugfix/996: strict="filter" doesn't work on spark dataframes #1001
• @williamjamir made their first contribution in unpin pandas-stubs version #1000
• @kr-hansen made their first contribution in Add Union support to check_types: Bugfix/977 #995
• @joepatol made their first contribution in Bugfix/997 #1017
• @erichamers made their first contribution in Unpinning ray from requirements-dev.txt #1052
• @brl0 made their first contribution in Small fix to example #1083
• @HenriqueAJNB made their first contribution in fix(dtypes.py): correction at function is_numeric docstring #1100

Full Changelog: v0.13.4...v0.14.0

---

This discussion was created from the release v0.14.0: Pandera Internals Rewrite [phase 1].

[a-recknagel]

Congratulations on the release! By the size of the diffs, this must have been a massive task. Thanks a lot for the great and diligent work 🎉

I'm looking forward to all the features that can now much more realistically be implemented, like #252 😁