-
Notifications
You must be signed in to change notification settings - Fork 217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
memtier_benchmark doest not exit as expected #94
Comments
Following up on this issue... any updates? |
add missing bufferevent_disable for connections that complete to send new requests
Seems like the fix was removed from the mainline again. Any reason for this or was this by accident? #106 |
Yes it did not exit for me in cluster-mode when benchmarking against a remote host, but did exit when benchmarking against localhost. |
With the same command-line arguments? if not, can you share please the command line that you used? |
It did not exit with the default arguments except for |
@lc0305 can you please, run it with |
@Teddyhoang not sure I got it... it seems that in your example the test exit as expected, no? |
Hi there, I'm running into the same problem with it didn't exit running cluster mode. It stuck at 100%. Any idea on what causes it and potential fixes? Thanks! |
I'm pulling the redislabs/memtier_benchmark:edge image, and looks like it's 2.0.0
|
@molian6 can you run it with |
It's not stuck all the time, but about 30% of the runs. I managed to repro once with
The behavior I observed is it hung there for 10+ minutes, but it would finally return with the correct stats report. I obfuscated the endpoint, but they are just the 3 node ip and port endpoint, but following it the last 100 logs I got for
|
Maybe it's also worth mention that this happened more frequently when server drop the connections. It looks like to maybe some error handling cases we've missed.
|
Thanks, @molian6. I tried to reproduce it with no success. I'll try to see if playing with the server connection helps. |
When a connection is dropped, it needs to empty its pipeline of pending responses, so in case of reconnection, it won't hang waiting for old requests.
Co-authored-by: YaacovHazan <[email protected]>
Thanks for the quick fix! I'll patch the PR and verify if we see issues resolved. |
Patched your changes but still getting the same issue Connection error: Connection timed out |
Hi @shailymittal can you provide more details? is it fails constantly? command line & output with debug level can be useful |
I hit this today as well with a
(...continues higher than 101%) I'll see if I can repro with |
What's also annoying is there's no way to get memtier to exit without losing all my results when it gets into this state... (I'm outputting to json file.) Even if I try to let it gracefully close with SIGTERM, it won't write the results it gathered to file so the whole run is lost 😢 |
@DrEsteban Thanks for raising it. On which version did you run the test? Can you provide the other arguments in the command line? Regarding your second comment. We have already seen in the past a request to dump the results also in case the test failed, I think we can combine it with yours |
# memtier_benchmark --version
memtier_benchmark 2.1.0 I'm running the test against an Azure Enterprise Redis cluster with: memtier_benchmark --server=<server> --port=10000 --authenticate=<key> --hide-histogram --pipeline=10 --clients=256 --threads=32 --data-size=1024 --test-time=600 --key-maximum=3041187 --key-pattern=R:R --ratio=0:1 --distinct-client-seed --run-count=1 --json-out-file=<outfile> --cluster-mode Haven't had a chance to get the |
@YaacovHazan I have a debug log for you! But it's 77MB |
@DrEsteban thanks, do you want to try via mail / shared google drive? |
@YaacovHazan I shared these with you via Slack about a week ago, just FYI! |
In my test enviroment, I got the following outputs:
$ ./memtier_benchmark -s 192.168.111.126 -p 2001 -n 1 -c 1 -t 2 -d 20000 --cluster-mode
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 0 secs] 2 threads: 2 ops, 6557 (avg: 6557) ops/sec, 125.38MB/sec (avg: 125.38MB/sec), 0.39 (avg: 0.39) msec latency
It will hang here like forever. By reading the code, it seems it should exit and output like this:
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 0 secs] 0 threads: 2 ops, 0 (avg: 6756) ops/sec, 0.00KB/sec (avg: 129.20MB/sec), 0.00 (avg: 0.37) msec latency
2 Threads
1 Connections per thread
1 Requests per client
ALL STATS
Type Ops/sec Hits/sec Misses/sec MOVED/sec ASK/sec Latency KB/sec
Sets 6779.66 --- --- 0.00 0.00 0.37200 132746.29
Gets 0.00 0.00 0.00 0.00 0.00 0.00000 0.00
Waits 0.00 --- --- --- --- 0.00000 ---
Totals 6779.66 0.00 0.00 0.00 0.00 0.37200 132746.29
Request Latency Distribution
Type <= msec Percent
SET 0.350 50.00
SET 0.390 100.00
Why? My test environment is cluster mode.
The text was updated successfully, but these errors were encountered: