Skip to content

Commit

Permalink
[KYUUBI#4935] More than target num of executors may survive after Fin…
Browse files Browse the repository at this point in the history
…alStageResourceManager did kill

### _Why are the changes needed?_
When FinalStageResourceManager chooses executors to be killed, it may add dead executors to the kill list.
This will leave more than target num of executors survived and cause resource waste.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4936 from zhouyifan279/kill-executor.

Closes #4936

2aaa84c [zhouyifan279] [KYUUBI#4935][Improvement] More than target num of executors may survive after FinalStageResourceManager did kill

Authored-by: zhouyifan279 <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
  • Loading branch information
zhouyifan279 authored and pan3793 committed Jun 8, 2023
1 parent 787028e commit d8415aa
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ case class FinalStageResourceManager(session: SparkSession)

// Evict the rest executors according to the shuffle block size
executorToBlockSize.toSeq.sortBy(_._2).foreach { case (id, _) =>
if (executorIdsToKill.length < expectedNumExecutorToKill) {
if (executorIdsToKill.length < expectedNumExecutorToKill && existedExecutors.contains(id)) {
executorIdsToKill.append(id)
}
}
Expand Down

0 comments on commit d8415aa

Please sign in to comment.