not all jars getting added to classpath in spark 3.5.0

[paulpaul1076]

I have a pod with jars, and the jars are located at path "/dependencies/test_jars". This path is mounted into pods that run spark.

For some odd reason, only 2 jars are getting picked up from there, yet I have many more jars there.

Here's my yml for running the spark application:

apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
  name: spark-action-event-streaming45
  namespace: spark
spec:
  version: "1.0"
  mainApplicationFile: s3a://obs-zdp-warehouse-stage-mz/firestarter-1.0.0.jar
  mainClass: com.ooo.firestarter.spark.streaming.job.impl.de.cs.v3.ClickstreamV3AppActionEventToIcebergStreamingJob
  args:
    - "env=stage"
    - "token=s.***"
    - "event_date=2023-08-17"
  sparkImage:
    productVersion: 3.5.0
    pullPolicy: IfNotPresent
  mode: cluster
  sparkConf:
    spark.driver.extraClassPath: /dependencies/test_jars/*:/stackable/spark/extra-jars/*
    spark.executor.extraClassPath: /dependencies/test_jars/*:/stackable/spark/extra-jars/*
    spark.hadoop.fs.s3.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
    spark.hadoop.fs.s3a.access.key: ***
    spark.hadoop.fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
    spark.hadoop.fs.s3a.bucket.obs-de-cicd-mz.access.key: ***
    spark.hadoop.fs.s3a.bucket.obs-de-cicd-mz.secret.key: ***
    spark.hadoop.fs.s3a.bucket.obs-de-export-mz.access.key: ***
    spark.hadoop.fs.s3a.bucket.obs-de-export-mz.secret.key: ***
    spark.hadoop.fs.s3a.bucket.obs-de-temporary-mz.access.key: ***
    spark.hadoop.fs.s3a.bucket.obs-de-temporary-mz.secret.key: ***
    spark.hadoop.fs.s3a.bucket.obs-zdp-warehouse-stage-mz.access.key: ***
    spark.hadoop.fs.s3a.bucket.obs-zdp-warehouse-stage-mz.secret.key: ***
    spark.hadoop.fs.s3a.endpoint: ***
    spark.hadoop.fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
    spark.hadoop.fs.s3a.secret.key: ***
    spark.kubernetes.memoryOverheadFactor: '0.2'
    spark.sql.ansi.enabled: 'true'
    spark.sql.catalog.iceberg_catalog.io-impl: org.apache.iceberg.aws.s3.S3FileIO
    spark.sql.catalog.iceberg_catalog.s3.access.key: ***
    spark.sql.catalog.iceberg_catalog.s3.delete-enabled: 'false'
    spark.sql.catalog.iceberg_catalog.s3.endpoint: ***
    spark.sql.catalog.iceberg_catalog.s3.path-style-access: 'true'
    spark.sql.catalog.iceberg_catalog.s3.secret.key: ***
    spark.sql.catalog.iceberg_catalog.s3a.path-style-access: 'true'
    spark.sql.catalog.iceberg_catalog.type: hive
    spark.sql.catalog.iceberg_catalog.uri: ***
    spark.sql.catalog.iceberg_catalog.warehouse: s3a://obs-zdp-warehouse-stage-mz/
    spark.sql.catalog.iceberg_catalog: org.apache.iceberg.spark.SparkCatalog
    spark.sql.catalogImplementation: hive
    spark.sql.debug: 'false'
    spark.sql.defaultCatalog: iceberg_catalog
    spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
    spark.sql.storeAssignmentPolicy: ANSI
    spark.ui.proxyBase: /sparkui/spark-example
    spark.ui.proxyRedirectUri: /


  logFileDirectory:
    s3:
      prefix: eventlogs/
      bucket:
        inline:
          bucketName: obs-zdp-warehouse-stage-mz
          connection:
            inline:
              host: ***
              tls:
                verification:
                  server:
                    caCert:
                      webPki: {}
              port: 443
              accessStyle: Path
              credentials:
                secretClass: obs-s3-credentials-secret
  volumes:
    - name: job-deps
      persistentVolumeClaim:
        claimName: spark-pvc
  env:
    - name: AWS_ACCESS_KEY_ID
      value: "***"
    - name: AWS_SECRET_ACCESS_KEY
      value: "***"

  driver:
    config:
      resources:
        cpu:
          min: '250m'
          max: "1"
        memory:
          limit: '8Gi'
      volumeMounts:
      - name: job-deps
        mountPath: /dependencies/test_jars
  executor:
    replicas: 3
    config:
      resources:
        cpu:
          min: '500m'
          max: "3"
        memory:
          limit: '4Gi'
      volumeMounts:
      - name: job-deps
        mountPath: /dependencies/test_jars

Here's the list of jars that I have in "/dependencies/test_jars":

root@testmount-lib:/# ls /dependencies/test_jars
iceberg-aws-1.4.2.jar                     logback-classic-1.2.11.jar  simpleclient_common-0.16.0.jar      simpleclient_pushgateway-0.16.0.jar    simpleclient_tracer_otel-0.16.0.jar        spark-sql-kafka-0-10_2.12-3.5.0.jar
iceberg-spark-runtime-3.5_2.12-1.4.2.jar  simpleclient-0.16.0.jar     simpleclient_httpserver-0.16.0.jar  simpleclient_tracer_common-0.16.0.jar  simpleclient_tracer_otel_agent-0.16.0.jar

And here's what I get when I print out classpath:
/dependencies/test_jars/iceberg-spark-runtime-3.5_2.12-1.4.2.jar:/dependencies/test_jars/iceberg-aws-bundle-1.4.2.jar:/stackable/spark/extra-jars/woodstox-core-6.5.1.jar:/stackable/spark/extra-jars/jackson-dataformat-xml-2.15.2.jar:/stackable/spark/extra-jars/stax2-api-4.2.1.jar:/opt/spark/conf/:/stackable/spark/jars/kubernetes-model-metrics-6.7.2.jar:/stackable/spark/jars/hadoop-client-api-3.3.4.jar:/stackable/spark/jars/tink-1.9.0.jar:/stackable/spark/jars/spark-kubernetes_2.12-3.5.0.jar:/stackable/spark/jars/spark-common-utils_2.12-3.5.0.jar:/stackable/spark/jars/py4j-0.10.9.7.jar:/stackable/spark/jars/jackson-core-asl-1.9.13.jar:/stackable/spark/jars/jackson-databind-2.15.2.jar:/stackable/spark/jars/spark-sql_2.12-3.5.0.jar:/stackable/spark/jars/json4s-ast_2.12-3.7.0-M11.jar:/stackable/spark/jars/pickle-1.3.jar:/stackable/spark/jars/hive-metastore-2.3.9.jar:/stackable/spark/jars/super-csv-2.2.0.jar:/stackable/spark/jars/hive-exec-2.3.9-core.jar:/stackable/spark/jars/datanucleus-rdbms-4.1.19.jar:/stackable/spark/jars/netty-transport-native-epoll-4.1.96.Final-linux-x86_64.jar:/stackable/spark/jars/JLargeArrays-1.5.jar:/stackable/spark/jars/joda-time-2.12.5.jar:/stackable/spark/jars/javax.jdo-3.2.0-m3.jar:/stackable/spark/jars/datanucleus-core-4.1.17.jar:/stackable/spark/jars/spark-mllib-local_2.12-3.5.0.jar:/stackable/spark/jars/hive-service-rpc-3.1.3.jar:/stackable/spark/jars/algebra_2.12-2.0.1.jar:/stackable/spark/jars/json4s-scalap_2.12-3.7.0-M11.jar:/stackable/spark/jars/netty-transport-4.1.96.Final.jar:/stackable/spark/jars/parquet-common-1.13.1.jar:/stackable/spark/jars/spark-launcher_2.12-3.5.0.jar:/stackable/spark/jars/spire-macros_2.12-0.17.0.jar:/stackable/spark/jars/snappy-java-1.1.10.3.jar:/stackable/spark/jars/jsr305-3.0.0.jar:/stackable/spark/jars/netty-handler-4.1.96.Final.jar:/stackable/spark/jars/jersey-container-servlet-2.40.jar:/stackable/spark/jars/flatbuffers-java-1.12.0.jar:/stackable/spark/jars/commons-collections4-4.4.jar:/stackable/spark/jars/breeze_2.12-2.1.0.jar:/stackable/spark/jars/avro-ipc-1.11.2.jar:/stackable/spark/jars/commons-io-2.13.0.jar:/stackable/spark/jars/jdo-api-3.0.1.jar:/stackable/spark/jars/parquet-encoding-1.13.1.jar:/stackable/spark/jars/paranamer-2.8.jar:/stackable/spark/jars/spark-mesos_2.12-3.5.0.jar:/stackable/spark/jars/zjsonpatch-0.3.0.jar:/stackable/spark/jars/lz4-java-1.8.0.jar:/stackable/spark/jars/netty-codec-4.1.96.Final.jar:/stackable/spark/jars/jersey-client-2.40.jar:/stackable/spark/jars/kubernetes-client-6.7.2.jar:/stackable/spark/jars/spire-platform_2.12-0.17.0.jar:/stackable/spark/jars/hk2-locator-2.6.1.jar:/stackable/spark/jars/kubernetes-client-api-6.7.2.jar:/stackable/spark/jars/commons-codec-1.16.0.jar:/stackable/spark/jars/RoaringBitmap-0.9.45.jar:/stackable/spark/jars/shims-0.9.45.jar:/stackable/spark/jars/mesos-1.4.3-shaded-protobuf.jar:/stackable/spark/jars/netty-codec-http2-4.1.96.Final.jar:/stackable/spark/jars/xz-1.9.jar:/stackable/spark/jars/oro-2.0.8.jar:/stackable/spark/jars/libthrift-0.12.0.jar:/stackable/spark/jars/kryo-shaded-4.0.2.jar:/stackable/spark/jars/arpack_combined_all-0.1.jar:/stackable/spark/jars/annotations-17.0.0.jar:/stackable/spark/jars/opencsv-2.3.jar:/stackable/spark/jars/spark-repl_2.12-3.5.0.jar:/stackable/spark/jars/netty-common-4.1.96.Final.jar:/stackable/spark/jars/scala-parser-combinators_2.12-2.3.0.jar:/stackable/spark/jars/curator-recipes-2.13.0.jar:/stackable/spark/jars/netty-transport-native-epoll-4.1.96.Final-linux-aarch_64.jar:/stackable/spark/jars/kubernetes-model-flowcontrol-6.7.2.jar:/stackable/spark/jars/spark-sketch_2.12-3.5.0.jar:/stackable/spark/jars/hive-shims-2.3.9.jar:/stackable/spark/jars/jaxb-runtime-2.3.2.jar:/stackable/spark/jars/arrow-format-12.0.1.jar:/stackable/spark/jars/arpack-3.0.3.jar:/stackable/spark/jars/log4j-core-2.20.0.jar:/stackable/spark/jars/kubernetes-model-autoscaling-6.7.2.jar:/stackable/spark/jars/kubernetes-model-rbac-6.7.2.jar:/stackable/spark/jars/datanucleus-api-jdo-4.2.4.jar:/stackable/spark/jars/hive-shims-common-2.3.9.jar:/stackable/spark/jars/hive-serde-2.3.9.jar:/stackable/spark/jars/rocksdbjni-8.3.2.jar:/stackable/spark/jars/netty-buffer-4.1.96.Final.jar:/stackable/spark/jars/janino-3.1.9.jar:/stackable/spark/jars/jpam-1.1.jar:/stackable/spark/jars/commons-lang3-3.12.0.jar:/stackable/spark/jars/jackson-module-scala_2.12-2.15.2.jar:/stackable/spark/jars/spark-sql-api_2.12-3.5.0.jar:/stackable/spark/jars/spark-network-shuffle_2.12-3.5.0.jar:/stackable/spark/jars/netty-all-4.1.96.Final.jar:/stackable/spark/jars/jersey-hk2-2.40.jar:/stackable/spark/jars/kubernetes-model-extensions-6.7.2.jar:/stackable/spark/jars/hk2-utils-2.6.1.jar:/stackable/spark/jars/istack-commons-runtime-3.0.8.jar:/stackable/spark/jars/kubernetes-model-apps-6.7.2.jar:/stackable/spark/jars/hadoop-shaded-guava-1.1.1.jar:/stackable/spark/jars/jackson-datatype-jsr310-2.15.2.jar:/stackable/spark/jars/arrow-memory-core-12.0.1.jar:/stackable/spark/jars/kubernetes-model-node-6.7.2.jar:/stackable/spark/jars/arrow-memory-netty-12.0.1.jar:/stackable/spark/jars/jersey-container-servlet-core-2.40.jar:/stackable/spark/jars/hive-jdbc-2.3.9.jar:/stackable/spark/jars/spark-unsafe_2.12-3.5.0.jar:/stackable/spark/jars/kubernetes-model-certificates-6.7.2.jar:/stackable/spark/jars/httpclient-4.5.14.jar:/stackable/spark/jars/jcl-over-slf4j-2.0.7.jar:/stackable/spark/jars/commons-cli-1.5.0.jar:/stackable/spark/jars/spire-util_2.12-0.17.0.jar:/stackable/spark/jars/netty-resolver-4.1.96.Final.jar:/stackable/spark/jars/jakarta.annotation-api-1.3.5.jar:/stackable/spark/jars/datasketches-java-3.3.0.jar:/stackable/spark/jars/httpcore-4.4.16.jar:/stackable/spark/jars/kubernetes-model-scheduling-6.7.2.jar:/stackable/spark/jars/jakarta.validation-api-2.0.2.jar:/stackable/spark/jars/jackson-core-2.15.2.jar:/stackable/spark/jars/hadoop-client-runtime-3.3.4.jar:/stackable/spark/jars/univocity-parsers-2.9.1.jar:/stackable/spark/jars/jersey-common-2.40.jar:/stackable/spark/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/stackable/spark/jars/netty-codec-http-4.1.96.Final.jar:/stackable/spark/jars/compress-lzf-1.1.2.jar:/stackable/spark/jars/kubernetes-model-events-6.7.2.jar:/stackable/spark/jars/metrics-json-4.2.19.jar:/stackable/spark/jars/commons-dbcp-1.4.jar:/stackable/spark/jars/spark-graphx_2.12-3.5.0.jar:/stackable/spark/jars/datasketches-memory-2.1.0.jar:/stackable/spark/jars/netty-transport-classes-kqueue-4.1.96.Final.jar:/stackable/spark/jars/jakarta.ws.rs-api-2.1.6.jar:/stackable/spark/jars/parquet-jackson-1.13.1.jar:/stackable/spark/jars/hive-cli-2.3.9.jar:/stackable/spark/jars/javolution-5.5.1.jar:/stackable/spark/jars/HikariCP-2.5.1.jar:/stackable/spark/jars/spark-core_2.12-3.5.0.jar:/stackable/spark/jars/netty-transport-native-unix-common-4.1.96.Final.jar:/stackable/spark/jars/hive-common-2.3.9.jar:/stackable/spark/jars/hive-storage-api-2.8.1.jar:/stackable/spark/jars/arrow-vector-12.0.1.jar:/stackable/spark/jars/curator-client-2.13.0.jar:/stackable/spark/jars/metrics-core-4.2.19.jar:/stackable/spark/jars/activation-1.1.1.jar:/stackable/spark/jars/commons-text-1.10.0.jar:/stackable/spark/jars/kubernetes-model-policy-6.7.2.jar:/stackable/spark/jars/cats-kernel_2.12-2.1.1.jar:/stackable/spark/jars/xbean-asm9-shaded-4.23.jar:/stackable/spark/jars/okio-1.15.0.jar:/stackable/spark/jars/scala-collection-compat_2.12-2.7.0.jar:/stackable/spark/jars/commons-pool-1.5.4.jar:/stackable/spark/jars/netty-transport-native-kqueue-4.1.96.Final-osx-x86_64.jar:/stackable/spark/jars/javassist-3.29.2-GA.jar:/stackable/spark/jars/scala-reflect-2.12.18.jar:/stackable/spark/jars/JTransforms-3.1.jar:/stackable/spark/jars/spark-streaming_2.12-3.5.0.jar:/stackable/spark/jars/log4j-1.2-api-2.20.0.jar:/stackable/spark/jars/aircompressor-0.25.jar:/stackable/spark/jars/scala-library-2.12.18.jar:/stackable/spark/jars/hive-shims-0.23-2.3.9.jar:/stackable/spark/jars/avro-1.11.2.jar:/stackable/spark/jars/commons-lang-2.6.jar:/stackable/spark/jars/antlr-runtime-3.5.2.jar:/stackable/spark/jars/kubernetes-model-admissionregistration-6.7.2.jar:/stackable/spark/jars/kubernetes-httpclient-okhttp-6.7.2.jar:/stackable/spark/jars/metrics-jvm-4.2.19.jar:/stackable/spark/jars/kubernetes-model-gatewayapi-6.7.2.jar:/stackable/spark/jars/kubernetes-model-core-6.7.2.jar:/stackable/spark/jars/zookeeper-3.6.3.jar:/stackable/spark/jars/jakarta.xml.bind-api-2.3.2.jar:/stackable/spark/jars/commons-math3-3.6.1.jar:/stackable/spark/jars/kubernetes-model-coordination-6.7.2.jar:/stackable/spark/jars/aopalliance-repackaged-2.6.1.jar:/stackable/spark/jars/commons-crypto-1.1.0.jar:/stackable/spark/jars/gson-2.2.4.jar:/stackable/spark/jars/leveldbjni-all-1.8.jar:/stackable/spark/jars/spark-hive_2.12-3.5.0.jar:/stackable/spark/jars/curator-framework-2.13.0.jar:/stackable/spark/jars/jackson-dataformat-yaml-2.15.2.jar:/stackable/spark/jars/json4s-jackson_2.12-3.7.0-M11.jar:/stackable/spark/jars/parquet-hadoop-1.13.1.jar:/stackable/spark/jars/okhttp-3.12.12.jar:/stackable/spark/jars/orc-mapreduce-1.9.1-shaded-protobuf.jar:/stackable/spark/jars/breeze-macros_2.12-2.1.0.jar:/stackable/spark/jars/threeten-extra-1.7.1.jar:/stackable/spark/jars/log4j-api-2.20.0.jar:/stackable/spark/jars/orc-shims-1.9.1.jar:/stackable/spark/jars/scala-xml_2.12-2.1.0.jar:/stackable/spark/jars/audience-annotations-0.5.0.jar:/stackable/spark/jars/hive-shims-scheduler-2.3.9.jar:/stackable/spark/jars/netty-handler-proxy-4.1.96.Final.jar:/stackable/spark/jars/spark-mllib_2.12-3.5.0.jar:/stackable/spark/jars/objenesis-3.3.jar:/stackable/spark/jars/kubernetes-model-resource-6.7.2.jar:/stackable/spark/jars/kubernetes-model-common-6.7.2.jar:/stackable/spark/jars/jersey-server-2.40.jar:/stackable/spark/jars/slf4j-api-2.0.7.jar:/stackable/spark/jars/lapack-3.0.3.jar:/stackable/spark/jars/spark-kvstore_2.12-3.5.0.jar:/stackable/spark/jars/logging-interceptor-3.12.12.jar:/stackable/spark/jars/chill_2.12-0.10.0.jar:/stackable/spark/jars/commons-compiler-3.1.9.jar:/stackable/spark/jars/minlog-1.3.0.jar:/stackable/spark/jars/scala-compiler-2.12.18.jar:/stackable/spark/jars/orc-core-1.9.1-shaded-protobuf.jar:/stackable/spark/jars/hive-beeline-2.3.9.jar:/stackable/spark/jars/netty-transport-classes-epoll-4.1.96.Final.jar:/stackable/spark/jars/jackson-annotations-2.15.2.jar:/stackable/spark/jars/spire_2.12-0.17.0.jar:/stackable/spark/jars/kubernetes-model-apiextensions-6.7.2.jar:/stackable/spark/jars/derby-10.14.2.0.jar:/stackable/spark/jars/log4j-slf4j2-impl-2.20.0.jar:/stackable/spark/jars/zookeeper-jute-3.6.3.jar:/stackable/spark/jars/snakeyaml-2.0.jar:/stackable/spark/jars/netty-transport-native-kqueue-4.1.96.Final-osx-aarch_64.jar:/stackable/spark/jars/stax-api-1.0.1.jar:/stackable/spark/jars/snakeyaml-engine-2.6.jar:/stackable/spark/jars/metrics-jmx-4.2.19.jar:/stackable/spark/jars/chill-java-0.10.0.jar:/stackable/spark/jars/avro-mapred-1.11.2.jar:/stackable/spark/jars/commons-compress-1.23.0.jar:/stackable/spark/jars/antlr4-runtime-4.9.3.jar:/stackable/spark/jars/hk2-api-2.6.1.jar:/stackable/spark/jars/kubernetes-model-storageclass-6.7.2.jar:/stackable/spark/jars/ST4-4.0.4.jar:/stackable/spark/jars/bonecp-0.8.0.RELEASE.jar:/stackable/spark/jars/hadoop-yarn-server-web-proxy-3.3.4.jar:/stackable/spark/jars/commons-logging-1.1.3.jar:/stackable/spark/jars/kubernetes-model-networking-6.7.2.jar:/stackable/spark/jars/jakarta.inject-2.6.1.jar:/stackable/spark/jars/transaction-api-1.1.jar:/stackable/spark/jars/json-1.8.jar:/stackable/spark/jars/kubernetes-model-discovery-6.7.2.jar:/stackable/spark/jars/jta-1.1.jar:/stackable/spark/jars/json4s-core_2.12-3.7.0-M11.jar:/stackable/spark/jars/kubernetes-model-batch-6.7.2.jar:/stackable/spark/jars/jakarta.servlet-api-4.0.3.jar:/stackable/spark/jars/jul-to-slf4j-2.0.7.jar:/stackable/spark/jars/parquet-column-1.13.1.jar:/stackable/spark/jars/jackson-mapper-asl-1.9.13.jar:/stackable/spark/jars/spark-hive-thriftserver_2.12-3.5.0.jar:/stackable/spark/jars/blas-3.0.3.jar:/stackable/spark/jars/metrics-graphite-4.2.19.jar:/stackable/spark/jars/stream-2.9.6.jar:/stackable/spark/jars/guava-14.0.1.jar:/stackable/spark/jars/netty-codec-socks-4.1.96.Final.jar:/stackable/spark/jars/spark-tags_2.12-3.5.0.jar:/stackable/spark/jars/libfb303-0.9.3.jar:/stackable/spark/jars/commons-collections-3.2.2.jar:/stackable/spark/jars/jline-2.14.6.jar:/stackable/spark/jars/hive-llap-common-2.3.9.jar:/stackable/spark/jars/osgi-resource-locator-1.0.3.jar:/stackable/spark/jars/spark-catalyst_2.12-3.5.0.jar:/stackable/spark/jars/jodd-core-3.5.2.jar:/stackable/spark/jars/spark-network-common_2.12-3.5.0.jar:/stackable/spark/jars/parquet-format-structures-1.13.1.jar:/stackable/spark/jars/spark-yarn_2.12-3.5.0.jar:/stackable/spark/jars/zstd-jni-1.5.5-4.jar:/stackable/spark/jars/ivy-2.5.1.jar:/stackable/spark/jars/azure-keyvault-core-1.0.0.jar:/stackable/spark/jars/azure-storage-7.0.1.jar:/stackable/spark/jars/hadoop-azure-3.3.4.jar:/stackable/spark/jars/aws-java-sdk-bundle-1.12.262.jar:/stackable/spark/jars/hadoop-aws-3.3.4.jar

Why does this happen?

[razvan]

Hey,
to better understand what is happening we need more context.

1. What is the actual error you are getting? I assume some sort of ClassNotFoundException, but I want to make sure.
2. Where is the listing you posted coming from : driver or executor? Are they the same ?
3. What happens if you explicitly add the jars from your /dependencies/test_jars folder to the [driver|executor].extraClassPath ?

[paulpaul1076]

1. Yes.
2. Driver
3. I did add them to my extraCLasPath as you can see in the yaml that I sent

I actually figured it out

So, I have a pod, called testmount-lib

and it has 2 paths:

/dependencies/jars (only 2 jars here)
/dependencies/test_jars (around 10 jars here)

In my yml for the spark job I was specifying /dependencies/test_jars, but the problem is that testmount-lib only exposes the /dependencies/jars path, so I can write any kind of path into spec.driver.config.volumeMounts.mountPath, the jars will still be taken from /dependencies/jars because this is what testmount-lib exposes.