Skip to content

Commit

Permalink
[CELEBORN-1625] Add parameter skipCompress for pushOrMergeData
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Add new parameters to the pushOrMergeData method for gluten

### Why are the changes needed?
Currently, in the scenario of Gluten Fallback, it is possible for a stage to have both Native and Java shuffles simultaneously. Since the ShuffleClient is a singleton, the compression settings for the ShuffleClient will only utilize either the Native or Java configuration. By adding new parameters, we aim to allow Gluten to control whether compression should be applied in this scenario.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
CI

Closes #2769 from kerwin-zk/CELEBORN-1625.

Authored-by: xiyu.zk <[email protected]>
Signed-off-by: zky.zhoukeyong <[email protected]>
  • Loading branch information
kerwin-zk authored and waitinfuture committed Sep 30, 2024
1 parent bae2309 commit 3b2e70c
Showing 1 changed file with 6 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -907,7 +907,8 @@ public int pushOrMergeData(
int length,
int numMappers,
int numPartitions,
boolean doPush)
boolean doPush,
boolean skipCompress)
throws IOException {
// mapKey
final String mapKey = Utils.makeMapKey(shuffleId, mapId, attemptId);
Expand Down Expand Up @@ -972,7 +973,7 @@ public int pushOrMergeData(
// increment batchId
final int nextBatchId = pushState.nextBatchId();

if (shuffleCompressionEnabled) {
if (shuffleCompressionEnabled && !skipCompress) {
// compress data
final Compressor compressor = compressorThreadLocal.get();
compressor.compress(data, offset, length);
Expand Down Expand Up @@ -1278,7 +1279,8 @@ public int pushData(
length,
numMappers,
numPartitions,
true);
true,
false);
}

@Override
Expand Down Expand Up @@ -1312,6 +1314,7 @@ public int mergeData(
length,
numMappers,
numPartitions,
false,
false);
}

Expand Down

0 comments on commit 3b2e70c

Please sign in to comment.