Does the Operator support multiple versions of Spark running at the same time?

[mmm-micro]

Hi there, I've been using the old Google Spark Operator and now looking at this one too.

One thing I'm interested in but was clear from reading the docs was if or how this operator supports running different versions of Spark Applications in the same k8s cluster.

E.g. we have many Spark apps running and it would be difficult to upgrade them all at the same time.

Ideally, we'd like an operator that allows us to run multiple versions of Spark at the same time without causing any issues. This would allow us to upgrade the Spark apps one at a time.

Is this possible with this Spark Operator?

[adwk67]

Thanks for your question @mmm-micro. As you can see from our integration tests (e.g. here), we run product images for different versions using the same operator. For Spark, we test using 3.4.1, 3.4.2, 3.5.0, 3.5.1 (the nightly tests use just 3.5.1), so, yes, it is possible for the operator to run products in different versions, but only for the ones that are supported.

[mmm-micro]

Thank you, I see how it works now. A new pod is created to do the Spark Submit and that Pod's version of spark should align with the Spark app you're running. That seems like a good way of doing it.

[adwk67]

Yes, that's correct: we build a spark-submit command based off the contents of the CRD and then pass that off to Spark, specifying which product image to use (which is used for the initiating job, the driver and executors).

[mmm-micro]

I found this example:
https://docs.stackable.tech/home/stable/spark-k8s/usage-guide/examples#_pyspark_externally_located_dataset_artifact_available_via_pvcvolume_mount

---
apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
  name: example-sparkapp-image
  namespace: default
spec:
  image: docker.stackable.tech/stackable/ny-tlc-report:0.1.0
  sparkImage:
    productVersion: 3.5.1
  mode: cluster
  mainApplicationFile: local:///stackable/spark/jobs/ny_tlc_report.py
  args:
    - "--input 's3a://nyc-tlc/trip data/yellow_tripdata_2021-07.csv'"
  deps:
    requirements:
      - tabulate==0.8.9
  sparkConf:
    "spark.hadoop.fs.s3a.aws.credentials.provider": "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider"
  job:
    config:
      resources:
        cpu:
          min: "1"
          max: "1"
        memory:
          limit: "1Gi"
  driver:
    config:
      resources:
        cpu:
          min: "1"
          max: "1500m"
        memory:
          limit: "1Gi"
  executor:
    replicas: 3
    config:
      resources:
        cpu:
          min: "1"
          max: "4"
        memory:
          limit: "2Gi"

Which makes it look like you can have a different image for the spark submit and the workers and executors.

So, I gave a simple example a go:

---
apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
  name: pyspark-pi-docker-hub-3.5.1
  namespace: spark-apps
spec:
  image: docker.io/spark:3.5.1
  sparkImage:
    productVersion: 3.5.1
  mode: cluster
  mainApplicationFile: local:///opt/spark/examples/src/main/python/pi.py
  driver:
    config:
      resources:
        cpu:
          min: "1"
          max: "2"
        memory:
          limit: "1Gi"
  executor:
    replicas: 1
    config:
      resources:
        cpu:
          min: "1"
          max: "2"
        memory:
          limit: "1Gi"
...

I was hoping it would just try to run the official Spark image's Pi app. But it fails on the driver:

pyspark-pi-docker-hub-3-5-1-53eea4904ae15e7a-driver   0/1     Init:Error   0          4s

Defaulted container "spark" out of: spark, job (init)
Error from server (BadRequest): container "spark" in pod "pyspark-pi-docker-hub-3-5-1-53eea4904ae15e7a-driver" is waiting to start: PodInitializing

Should something like this work, or do we have to use an image based off a stackable image?

We tend to include the Spark apps in the image rather than pulling from S3.

To use this Operator, would have to build off a Stackable base image if we still want to include our apps and deps in the images we run?

[adwk67]

Can you open a separate question for this and I can answer it there?