You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When this code is run using Spark version 3.1.2 I got an error
Caused by: org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '2065-10-6' in the new parser. You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string.
at org.apache.spark.sql.catalyst.util.DateTimeFormatterHelper$$anonfun$checkParsedDiff$1.applyOrElse(DateTimeFormatterHelper.scala:150)
The fix for this issue is to add the following lines of code
SparkConf conf = new SparkConf().setAppName("Spark In Action").setMaster("local[*]");
conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY");
To make it work in Spark 3 what kind of formatting is required?
The text was updated successfully, but these errors were encountered:
net.jgp.books.spark.ch03/src/main/java/net/jgp/books/spark/ch03/lab320_dataset_books_to_dataframe/CsvToDatasetBookToDataframeApp.java
Line 110 in 1362aa6
When this code is run using Spark version 3.1.2 I got an error
Caused by: org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '2065-10-6' in the new parser. You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string.
at org.apache.spark.sql.catalyst.util.DateTimeFormatterHelper$$anonfun$checkParsedDiff$1.applyOrElse(DateTimeFormatterHelper.scala:150)
The fix for this issue is to add the following lines of code
To make it work in Spark 3 what kind of formatting is required?
The text was updated successfully, but these errors were encountered: