Old Spark Streaming version support

[AliGouta]

Hello,

We currently have spark streaming applications compiled and released with version 2.3.0 and a target kubernetes cluster version set to 1.24. Do you think that the spark operator supports old versions ?

Thanks !

[sbernauer]

Hello @AliGouta,

thanks for your interest!
We are aware that migrating your Spark jobs to a new Spark version is a humongous task.
However, we do not explicitly support or plan on supporting older versions but it might work.

In case you are interested in giving it a try yourself, here is the relevant Dockerfile and definition of the versions to build. You might be able to swap out the Spark version.

For the Kubernetes version, besides it not being not officially supported, I think the current Stackable versions (23.4 and lower) should work, as I am not aware of the use of any fancy features that were added after K8s 1.24.

All the best,
Sebastian