SumArray not working on Databricks cluster

[arnaudframmery]

SumArray is currently not working on Databricks cluster

You can reproduce the error by executing the main file, the error should be :

Exception in thread "main" java.lang.ClassCastException: cannot assign instance of org.apache.spark.sql.catalyst.trees.TreePattern$ to field org.apache.spark.sql.catalyst.trees.TreePattern$.WITH_WINDOW_DEFINITION of type scala.Enumeration$Value in instance of org.apache.spark.sql.catalyst.trees.TreePattern$

This error maybe comes from the Aggregator object in the SumArray class.

=> Don't forget to change the path to scala-collection-compat_2.12-2.1.1.jar in the pom.xml file

[sehyod]

We have implemented the same logic in scala, in a notebook on databricks (test_aggregator) and it does work, meaning the logic is correct. The java code also works locally, the problem only occurs when trying to execute the java code on the cluster.

[OPeyrusse]

For the record, we solved this issue by not packing the Encoder from the local session before sending the SumArray function to the cluster.
Instead, we retrieve it from the current session using SparkSession#active.

We made all tests pass adapting the code of tests and of the functions to support more generic types:

• ArraySeq<_> -> Seq<_>
• assuming that the ids are numbers that we can cast to int.