Skip to content

Commit

Permalink
[CELEBORN-897] Set celeborn.network.memory.allocator.allowCache defau…
Browse files Browse the repository at this point in the history
…lt to false

### What changes were proposed in this pull request?
As title

### Why are the changes needed?
I tested 1.1T and 3.3T shuffle, as well as 3T TPCDS with thread cache on and off in the shared PooledByteBufAllocator and find no
difference:
| Benchmark    | Cache On | Cache Off|
| -------- | ------- |------- |
|1.1T Shuffle| 3.7min/1.9min   |3.7min/1.9min|
| 3.3T Shuffle| 12min/6.7min  |12min/6.2min|
| 3T TPCDS | 2645s |2644s|

And since the configuration has a big influence to the direct memory usage, see #1716 , it's very necessary to set the default value to false.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Manual test.

Closes #1817 from waitinfuture/897.

Authored-by: zky.zhoukeyong <[email protected]>
Signed-off-by: zky.zhoukeyong <[email protected]>
(cherry picked from commit 57fdbf0)
Signed-off-by: zky.zhoukeyong <[email protected]>
  • Loading branch information
waitinfuture committed Aug 15, 2023
1 parent 6725e74 commit 4233ea3
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -1244,7 +1244,7 @@ object CelebornConf extends Logging {
.version("0.3.1")
.doc("When false, globally disable thread-local cache in the shared PooledByteBufAllocator.")
.booleanConf
.createWithDefault(true)
.createWithDefault(false)

val NETWORK_MEMORY_ALLOCATOR_SHARE: ConfigEntry[Boolean] =
buildConf("celeborn.network.memory.allocator.share")
Expand Down

0 comments on commit 4233ea3

Please sign in to comment.