Skip to content

Commit

Permalink
[Bug] Long tail tasks in the Write Stage retry phase results in data …
Browse files Browse the repository at this point in the history
…loss.
  • Loading branch information
yl09099 committed Dec 19, 2024
1 parent 0481f21 commit 355f691
Showing 1 changed file with 3 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,9 @@ public void reportShuffleWriteFailure(
// Clear the metadata of the completed task, otherwise some of the stage's data will
// be lost.
shuffleManager.unregisterAllMapOutput(shuffleId);
// Need to clear the mapStatus twice to prevent partition data loss due to the
// long-tail task performed before the stage retry.
shuffleManager.unregisterAllMapOutput(shuffleId);
// Deregister the shuffleId corresponding to the Shuffle Server.
shuffleManager.getShuffleWriteClient().unregisterShuffle(appId, shuffleId);
shuffleServerWriterFailureRecord.setClearedMapTrackerBlock(true);
Expand Down

0 comments on commit 355f691

Please sign in to comment.