Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add arbitration lock time out to shared arbitrator #11376

Closed
wants to merge 1 commit into from

Conversation

tanjialiang
Copy link
Contributor

Current arbitrator lock does not enforce timeout, which might lead to arbitration operation's timeout not strictly enforced if waiting for too long for a lock. It can also prevent deadlock situation from completely hanging the system.

@tanjialiang tanjialiang requested a review from xiaoxmeng October 29, 2024 16:49
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 29, 2024
Copy link

netlify bot commented Oct 29, 2024

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 4a5a73d
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/6732a64e45f94300085e60bf

@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

velox/common/memory/SharedArbitrator.h Outdated Show resolved Hide resolved
velox/common/memory/SharedArbitrator.h Outdated Show resolved Hide resolved
velox/common/memory/SharedArbitrator.cpp Show resolved Hide resolved
velox/common/memory/SharedArbitrator.cpp Show resolved Hide resolved
@tanjialiang tanjialiang force-pushed the arb_timeout branch 2 times, most recently from 3c0f916 to c71904a Compare October 29, 2024 18:40
Copy link
Contributor

@xiaoxmeng xiaoxmeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tanjialiang nice change. Thanks!

@@ -1221,12 +1203,12 @@ void SharedArbitrator::freeCapacity(uint64_t bytes) {
if (FOLLY_UNLIKELY(bytes == 0)) {
return;
}
std::vector<ContinuePromise> resumes;
std::vector<ContinuePromise> globalArbitrationWaitResumes;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just use short names which is clear in the context? Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In freeCapacity() method scope there isn't any context mentioning the resume is for global arbitration. So I changed the name to indicate. This way I don't need to refer to other locations of the file to figure out this resume is for global arbitration anymore.

velox/common/memory/SharedArbitrator.cpp Outdated Show resolved Hide resolved
velox/common/memory/ArbitrationParticipant.h Show resolved Hide resolved
velox/common/memory/ArbitrationParticipant.cpp Outdated Show resolved Hide resolved
@@ -3935,5 +3933,90 @@ TEST_F(MockSharedArbitrationTest, concurrentArbitrationWithTransientRoots) {
}
controlThread.join();
}
} // namespace

TEST_F(MockSharedArbitrationTest, arbitrationOperationTimedLock) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test should be in ArbitrationParticipantTest? Also cover the two APIs: abort and reclaim?

In MockSharedArbirrationTest, let's have two test cases with one for reclaim lock timeout and one for abort lock time out? Thanks!

@tanjialiang tanjialiang force-pushed the arb_timeout branch 4 times, most recently from 2d0f96a to 81961c7 Compare November 5, 2024 18:26
@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@tanjialiang tanjialiang force-pushed the arb_timeout branch 6 times, most recently from ad7d8ed to dea0450 Compare November 6, 2024 02:15
@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

1 similar comment
@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

…#11376)

Summary:
Current arbitrator lock does not enforce timeout, which might lead to arbitration operation's timeout not strictly enforced if waiting for too long for a lock. It can also prevent deadlock situation from completely hanging the system.


Reviewed By: xiaoxmeng

Differential Revision: D65151919

Pulled By: tanjialiang
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D65151919

@facebook-github-bot
Copy link
Contributor

@tanjialiang merged this pull request in 3e93c07.

Copy link

Conbench analyzed the 1 benchmark run on commit 3e93c074.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported Merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants