From 4233ea398e43206a8abff45008237a9cdb8fe702 Mon Sep 17 00:00:00 2001 From: "zky.zhoukeyong" Date: Wed, 16 Aug 2023 00:26:39 +0800 Subject: [PATCH] [CELEBORN-897] Set celeborn.network.memory.allocator.allowCache default to false ### What changes were proposed in this pull request? As title ### Why are the changes needed? I tested 1.1T and 3.3T shuffle, as well as 3T TPCDS with thread cache on and off in the shared PooledByteBufAllocator and find no difference: | Benchmark | Cache On | Cache Off| | -------- | ------- |------- | |1.1T Shuffle| 3.7min/1.9min |3.7min/1.9min| | 3.3T Shuffle| 12min/6.7min |12min/6.2min| | 3T TPCDS | 2645s |2644s| And since the configuration has a big influence to the direct memory usage, see https://github.com/apache/incubator-celeborn/pull/1716 , it's very necessary to set the default value to false. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual test. Closes #1817 from waitinfuture/897. Authored-by: zky.zhoukeyong Signed-off-by: zky.zhoukeyong (cherry picked from commit 57fdbf08c293534ea8b62245d1172225f41d8be2) Signed-off-by: zky.zhoukeyong --- .../main/scala/org/apache/celeborn/common/CelebornConf.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala index 8a351d493fa..e08517f6003 100644 --- a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala +++ b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala @@ -1244,7 +1244,7 @@ object CelebornConf extends Logging { .version("0.3.1") .doc("When false, globally disable thread-local cache in the shared PooledByteBufAllocator.") .booleanConf - .createWithDefault(true) + .createWithDefault(false) val NETWORK_MEMORY_ALLOCATOR_SHARE: ConfigEntry[Boolean] = buildConf("celeborn.network.memory.allocator.share")