Skip to content

Commit

Permalink
Improve the configuration document (apache#7334)
Browse files Browse the repository at this point in the history
  • Loading branch information
@beliefer
beliefer authored Sep 24, 2024
1 parent 95334d5 commit 2529c2e
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions docs/Configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@ You can add these configurations into spark-defaults.conf to enable or disable t

| Parameters | Description | Recommend Setting |
|--------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
| spark.driver.extraClassPath | To add Gluten Plugin jar file in Spark Driver | /path/to/jar_file |
| spark.executor.extraClassPath | To add Gluten Plugin jar file in Spark Executor | /path/to/jar_file |
| spark.executor.memory | To set up how much memory to be used for Spark Executor. | |
| spark.memory.offHeap.size | To set up how much memory to be used for Java OffHeap.<br /> Please notice Gluten Plugin will leverage this setting to allocate memory space for native usage even offHeap is disabled. <br /> The value is based on your system and it is recommended to set it larger if you are facing Out of Memory issue in Gluten Plugin | 30G |
| spark.sql.sources.useV1SourceList | Choose to use V1 source | avro |
| spark.sql.join.preferSortMergeJoin | To turn off preferSortMergeJoin in Spark | false |
| spark.driver.extraClassPath | Gluten Plugin jar file to prepend to the classpath of the driver. | /path/to/jar_file |
| spark.executor.extraClassPath | Gluten Plugin jar file to prepend to the classpath of executors. | /path/to/jar_file |
| spark.executor.memory | Amount of memory to use per executor process, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g). | |
| spark.memory.offHeap.size | The absolute amount of memory which can be used for off-heap allocation, in bytes unless otherwise specified.<br /> Note: Gluten Plugin will leverage this setting to allocate memory space for native usage even offHeap is disabled. <br /> The value is based on your system and it is recommended to set it larger if you are facing Out of Memory issue in Gluten Plugin | 30G |
| spark.sql.sources.useV1SourceList | A comma-separated list of data source short names or fully qualified data source implementation class names for which Data Source V2 code path is disabled. <br /> Note: Please use V1 source for avro. | avro |
| spark.sql.join.preferSortMergeJoin | When true, prefer sort merge join over shuffled hash join. <br /> Note: Please turn off preferSortMergeJoin. | false |
| spark.plugins | To load Gluten's components by Spark's plug-in loader | org.apache.gluten.GlutenPlugin |
| spark.shuffle.manager | To turn on Gluten Columnar Shuffle Plugin | org.apache.spark.shuffle.sort.ColumnarShuffleManager |
| spark.gluten.enabled | Enable Gluten, default is true. Just an experimental property. Recommend to enable/disable Gluten through the setting for `spark.plugins`. | true |
Expand Down

0 comments on commit 2529c2e

Please sign in to comment.