Jars dependencies for Delta (4.0.0rc1) using Pyspark (4.0.0preview2) ? --> Failing when save the df

[Y3NK]

Dear Community, does any of you was able to get a working Spark Session (4.0.0preview2) with Delta 4.0.0rc1 ?

I am in an air-gap server, so I need to bring manually the needed JARS (which is really fun!), but I am not able to get a working SparkSession able to write Spark dataFrame to delta files ...

The initializing part go well, but the
sdf.write.format("delta").save("sdf")
or variation with sd.parquet, full path, full path with .delta
failed every time (error bellow...)

Details:

Java Home | /opt/jdk-17.0.12+7
Java Version | 17.0.12 (Microsoft)
Scala Version | version 2.13.14

Instanciate:

my_session_spark = (SparkSession.builder 
        .appName(name) 
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") 
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

        .config("spark.jars",list_jar) 
        .getOrCreate()
    )

with list of jars:

• /JARS_pyspark/scala-library-2.13.13.jar,
• /JARS_pyspark/spark-token-provider-kafka-0-10_2.13-4.0.0-preview2.jar,
• /JARS_pyspark/spark-sql_2.13-4.0.0-preview2.jar,
• /JARS_pyspark/hadoop-common-3.4.0 (1).jar,
• /JARS_pyspark/spark-tags_2.13-4.0.0-preview2.jar,
• /JARS_pyspark/spark-catalyst_2.13-4.0.0-preview2.jar,
• /JARS_pyspark/spark-core_2.13-4.0.0-preview2.jar,
• /JARS_pyspark/aws-java-sdk-1.12.262.jar,
• /JARS_pyspark/spark-sql-kafka-0-10_2.13-4.0.0-preview2.jar,
• /JARS_pyspark/kafka-clients-3.8.0.jar,
• /JARS_pyspark/spark-connect_2.13-4.0.0-preview2.jar,
• /JARS_pyspark/commons-pool2-2.12.0.jar,
• /JARS_pyspark/hadoop-aws-3.4.0 (1).jar,
• /JARS_pyspark/spark-streaming_2.13-4.0.0-preview2.jar,

Mainly these ones:

• /JARS_pyspark/antlr4-runtime-4.13.2.jar,
• /JARS_pyspark/delta-spark_2.13-4.0.0rc1.jar,
• /JARS_pyspark/delta-storage-s3-dynamodb-4.0.0rc1.jar,
• /JARS_pyspark/delta-storage-4.0.0rc1.jar,
• /JARS_pyspark/delta-contribs_2.13-4.0.0rc1-javadoc.jar

Does anyone have a clue about which JAR could be wrong version / missing ?
Many thanks for your help :-)

But I got the error (which usually is related to a bad .JAR)

Py4JJavaError: An error occurred while calling o72.save.
: com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.delta.DeltaConfigs$
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2083)
at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4898)
at org.apache.spark.sql.delta.DeltaLog$.getDeltaLogFromCache$1(DeltaLog.scala:877)
at org.apache.spark.sql.delta.DeltaLog$.apply(DeltaLog.scala:887)
at org.apache.spark.sql.delta.DeltaLog$.forTable(DeltaLog.scala:763)
at org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:190)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:53)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:77)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:86)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$2(QueryExecution.scala:126)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$6(SQLExecution.scala:155)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:267)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:118)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:742)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId0(SQLExecution.scala:74)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:222)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:126)
at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:608)
at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$eagerlyExecute$1(QueryExecution.scala:125)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$3.applyOrElse(QueryExecution.scala:140)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$3.applyOrElse(QueryExecution.scala:135)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:470)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:86)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:470)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:37)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:330)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:326)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:37)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:37)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:446)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:135)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:100)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:157)
at org.apache.spark.sql.internal.DataFrameWriterImpl.runCommand(DataFrameWriterImpl.scala:614)
at org.apache.spark.sql.internal.DataFrameWriterImpl.saveToV1Source(DataFrameWriterImpl.scala:271)
at org.apache.spark.sql.internal.DataFrameWriterImpl.saveInternal(DataFrameWriterImpl.scala:233)
at org.apache.spark.sql.internal.DataFrameWriterImpl.save(DataFrameWriterImpl.scala:117)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.delta.DeltaConfigs$
at org.apache.spark.sql.delta.InitialSnapshot.(Snapshot.scala:584)
at org.apache.spark.sql.delta.SnapshotManagement.$anonfun$getSnapshotForLogSegmentInternal$5(SnapshotManagement.scala:1104)
at scala.Option.getOrElse(Option.scala:201)
at org.apache.spark.sql.delta.SnapshotManagement.getSnapshotForLogSegmentInternal(SnapshotManagement.scala:1102)
at org.apache.spark.sql.delta.SnapshotManagement.getSnapshotForLogSegmentInternal$(SnapshotManagement.scala:1085)
at org.apache.spark.sql.delta.DeltaLog.getSnapshotForLogSegmentInternal(DeltaLog.scala:74)
at org.apache.spark.sql.delta.SnapshotManagement.getUpdatedSnapshot(SnapshotManagement.scala:1065)
at org.apache.spark.sql.delta.SnapshotManagement.getUpdatedSnapshot$(SnapshotManagement.scala:1056)
at org.apache.spark.sql.delta.DeltaLog.getUpdatedSnapshot(DeltaLog.scala:74)
at org.apache.spark.sql.delta.SnapshotManagement.$anonfun$getSnapshotAtInit$2(SnapshotManagement.scala:634)
at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:168)
at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:166)
at org.apache.spark.sql.delta.DeltaLog.recordFrameProfile(DeltaLog.scala:74)
at org.apache.spark.sql.delta.SnapshotManagement.$anonfun$getSnapshotAtInit$1(SnapshotManagement.scala:624)
at org.apache.spark.sql.delta.SnapshotManagement.withSnapshotLockInterruptibly(SnapshotManagement.scala:78)
at org.apache.spark.sql.delta.SnapshotManagement.withSnapshotLockInterruptibly$(SnapshotManagement.scala:75)
at org.apache.spark.sql.delta.DeltaLog.withSnapshotLockInterruptibly(DeltaLog.scala:74)
at org.apache.spark.sql.delta.SnapshotManagement.getSnapshotAtInit(SnapshotManagement.scala:624)
at org.apache.spark.sql.delta.SnapshotManagement.getSnapshotAtInit$(SnapshotManagement.scala:623)
at org.apache.spark.sql.delta.DeltaLog.getSnapshotAtInit(DeltaLog.scala:74)
at org.apache.spark.sql.delta.SnapshotManagement.$init$(SnapshotManagement.scala:69)
at org.apache.spark.sql.delta.DeltaLog.(DeltaLog.scala:80)
at org.apache.spark.sql.delta.DeltaLog$.$anonfun$apply$4(DeltaLog.scala:865)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:386)
at org.apache.spark.sql.delta.DeltaLog$.$anonfun$apply$3(DeltaLog.scala:860)
at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:168)
at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:166)
at org.apache.spark.sql.delta.DeltaLog$.recordFrameProfile(DeltaLog.scala:663)
at org.apache.spark.sql.delta.metering.DeltaLogging.$anonfun$recordDeltaOperationInternal$1(DeltaLogging.scala:136)
at com.databricks.spark.util.DatabricksLogging.recordOperation(DatabricksLogging.scala:128)
at com.databricks.spark.util.DatabricksLogging.recordOperation$(DatabricksLogging.scala:117)
at org.apache.spark.sql.delta.DeltaLog$.recordOperation(DeltaLog.scala:663)
at org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperationInternal(DeltaLogging.scala:135)
at org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:125)
at org.apache.spark.sql.delta.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:115)
at org.apache.spark.sql.delta.DeltaLog$.recordDeltaOperation(DeltaLog.scala:663)
at org.apache.spark.sql.delta.DeltaLog$.createDeltaLog$1(DeltaLog.scala:859)
at org.apache.spark.sql.delta.DeltaLog$.$anonfun$apply$5(DeltaLog.scala:878)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4903)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)
... 51 more
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchMethodError: 'void org.apache.spark.sql.Column.(org.apache.spark.sql.catalyst.expressions.Expression)' [in thread "Thread-4"]
at org.apache.spark.sql.delta.stats.DataSkippingReader$.col(DataSkippingReader.scala:173)
at org.apache.spark.sql.delta.stats.DataSkippingReader$.(DataSkippingReader.scala:177)
at org.apache.spark.sql.delta.DeltaConfigsBase.$init$(DeltaConfig.scala:529)
at org.apache.spark.sql.delta.DeltaConfigs$.(DeltaConfig.scala:798)
... 94 more

[Y3NK]

It seems that indeed versions are causing issues. I had to revert to pyspark==4.0.0dev1. Because deltaLake4 is currently built on dev1.