[SPARK-47618][CORE] Use Magic Committer for all S3 buckets by default

Author: dongjoon-hyun

core/src/main/scala/org/apache/spark/SparkContext.scala

@@ -423,6 +423,10 @@ class SparkContext(config: SparkConf) extends Logging {
     if (!_conf.contains("spark.app.name")) {
       throw new SparkException("An application name must be set in your configuration")
     }
+    // Enable Magic Committer by default for all S3 buckets if hadoop-cloud module exists
+    if (Utils.classIsLoadable("org.apache.spark.internal.io.cloud.PathOutputCommitProtocol")) {
+      conf.setIfMissing("spark.hadoop.fs.s3a.bucket.*.committer.magic.enabled", "true")
+    }
     // This should be set as early as possible.
     SparkContext.fillMissingMagicCommitterConfsIfNeeded(_conf)
 

docs/core-migration-guide.md

@@ -24,6 +24,8 @@ license: |
 
 ## Upgrading from Core 3.5 to 4.0
 
+- Since Spark 4.0, Spark uses Apache Hadoop Magic Committer for all S3 buckets by default. To restore the behavior before Spark 4.0, you can set `spark.hadoop.fs.s3a.bucket.*.committer.magic.enabled=false`.
+
 - Since Spark 4.0, Spark migrated all its internal reference of servlet API from `javax` to `jakarta` 
 
 - Since Spark 4.0, Spark will roll event logs to archive them incrementally. To restore the behavior before Spark 4.0, you can set `spark.eventLog.rolling.enabled` to `false`.