Feature: Possibility to process PySpark DataFrames?

[ZlaTanskY]

Task: Should we have a possibility to process PySpark DataFrames?

Currently at Telenet there is a use case in which they use PySpark DataFrames and they would like to use the cobra preprocessing for creating their model. Uncertain that this is currently possible, this issue is created.

[sandervh14]

Hi Jano!

We have a branch spark-cobra that was once created for that.
You can try out (or can tell the person who contacted you for this) to try out if that branch can do the work.
Word of warning: this branch once was created to transform dataframes into the target encoding etc needed for the PIGs etc, but it is no longer up to date with our dev/main branch. It may still do the work needed, though. To be tried.

Sander