Production tuning #17
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Production setup is all contained within 1 pod, so everything is on localhost (same network namespace).
Some of these queries can take 2min+, during which time the connection is consumed. Fail quickly to allow more overall throughput, at the cost of a loss of correctness in the edge cases (edits will be skipped). Note: The queries are usually slow on accounts with lots of edits e.g. bots, which likely are not going to be reverted anyway. Sample timings (from botng, which uses the same queries): name P95(duration_ms) AVG(duration_ms) loader.LoadDistinctPagesCount 18,179.03402 3,005.03574 loader.LoadUserWarnsCount 572.02909 127.03932 loader.LoadPageRecentRevertCount 253.82347 66.91965 loader.LoadPageRevision 163.07911 106.22102 loader.LoadPageMetadata 104.4249 179.18445 loader.LoadUserEditCount 84.84196 11.34932 loader.LoadPageRecentEditCount 9.95227 6.23804 Example edits (also from botng, but shows the challenge) UUID duration_ms Related username 6f9b2f64-bced-4a2f-af1e-5f02bc3b4218 174,505.79937 InternetArchiveBot 94141464-459e-48cf-9901-9a9161fae17c 168,008.14787 AnomieBOT 05e3e528-3f36-493f-91d6-fb7e9aaaaa93 47,951.95634 EmausBot 89f2ef9c-1ec3-425e-b667-a92476a358cf 13,345.4171 Plantdrew 68733457-aafe-426f-b91c-cd85e4c733a6 5,618.97978 Dimadick
|
Query limits are working Sample timings (from botng, which uses the same queries) - user distinct pages count on top: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This will impact the scoring due to the user distinct pages & recent edit logic changes.
It will also prefer dropping edits, rather than hogging connections, which should improve overall throughput at the cost of large users.