Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RATIS-2146 Fixed possible issues caused by concurrent deletion and election when member changes #1140

Merged
merged 1 commit into from
Sep 3, 2024
Merged

RATIS-2146 Fixed possible issues caused by concurrent deletion and election when member changes #1140

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions ratis-server/src/main/java/org/apache/ratis/server/impl/RaftServerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
*/
package org.apache.ratis.server.impl;

import java.util.concurrent.CountDownLatch;
import org.apache.ratis.client.impl.ClientProtoUtils;
import org.apache.ratis.conf.RaftProperties;
import org.apache.ratis.metrics.Timekeeper;
Expand Down Expand Up @@ -229,6 +230,7 @@ public long[] getFollowerNextIndices() {
private final RaftServerJmxAdapter jmxAdapter = new RaftServerJmxAdapter(this);
private final LeaderElectionMetrics leaderElectionMetrics;
private final RaftServerMetricsImpl raftServerMetrics;
private final CountDownLatch closeFinishedLatch = new CountDownLatch(1);

// To avoid append entry before complete start() method
// For example, if thread1 start(), but before thread1 startAsFollower(), thread2 receive append entry
Expand Down Expand Up @@ -463,6 +465,13 @@ void groupRemove(boolean deleteDirectory, boolean renameDirectory) {
/* Shutdown is triggered here inorder to avoid any locked files. */
state.getStateMachineUpdater().setRemoving();
close();
try {
closeFinishedLatch.await();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CountDownLatch may not help. When close() returns, it must past the line closeFinishedLatch.countDown();. The new code seems the same as the existing code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

In fact, there is a difference because lifeCycle.checkStateAndClose is a CAS (Compare-And-Swap) operation, meaning only the thread that successfully performs the CAS can execute this function.

In our reproduced case, the election thread successfully performed the CAS to set the state to closing and was executing the shutdown code inside. Then, when the thread responsible for deleting the raftgroup called this function and found that the state was closing, it essentially did nothing and returned to delete the file directory, which subsequently caused an error in StateMachineUpdater.

I checked the contents of this lambda expression, and except for FollowerState and LeaderElection which are asynchronous, the rest should be synchronous, including the StateMachineUpdater that we discovered this time. Therefore, after adding this countDownLatch, at least the error we found will not occur again.

As for whether it's necessary to also synchronize the waiting for FollowerState and LeaderElection, I think it could be either way. Even if they clean up asynchronously, they will ultimately just transition to Leader, and the underlying shutdownLeaderState and state.close can prevent any impact from occurring.

Additionally, I found that we can't directly add a synchronized signature to the close function, because this lambda function will still only be executed by the thread that successfully performs the CAS.

Overall, I feel that the current changes should not introduce any side effects. What do you think? @szetszwo

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OneSizeFitsQuorum , thanks for the detailed explanation! I understand the change now.

} catch (InterruptedException e) {
Thread.currentThread().interrupt();
LOG.warn("{}: Waiting closing interrupted, will not continue to remove group locally", getMemberId());
return;
}
getStateMachine().event().notifyGroupRemove();
if (deleteDirectory) {
for (int i = 0; i < FileUtils.NUM_ATTEMPTS; i ++) {
Expand Down Expand Up @@ -541,6 +550,7 @@ public void close() {
} catch (Exception e) {
LOG.warn(getMemberId() + ": Failed to shutdown serverExecutor", e);
}
closeFinishedLatch.countDown();
});
}

Expand Down
Loading