You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When benchmarking (#13) against an echo server, the client will saturate at a point where it is writing messages faster than it is able to read them back from the server. When the client's socket read buffer starts to fill up, the echo server's socket write buffer will also fill up, and the echo server will stop reading from the socket. Once the client's socket write buffer also fills up, the client will end up in a deadlock situation where it fails with a write timeout:
Kontena::Websocket::TimeoutError: write timeout after 5.0s
/home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client/connection.rb:25:in `wait_socket_writable!'
/home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client/connection.rb:92:in `rescue in nonblocking_timeout'
/home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client/connection.rb:85:in `nonblocking_timeout'
/home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client/connection.rb:116:in `write'
/home/kontena/kontena/kontena-websocket-client/vendor/bundle/ruby/2.3.0/gems/websocket-driver-kontena-0.6.5/lib/websocket/driver/hybi.rb:228:in `send_frame'
/home/kontena/kontena/kontena-websocket-client/vendor/bundle/ruby/2.3.0/gems/websocket-driver-kontena-0.6.5/lib/websocket/driver/hybi.rb:191:in `frame'
/home/kontena/kontena/kontena-websocket-client/vendor/bundle/ruby/2.3.0/gems/websocket-driver-kontena-0.6.5/lib/websocket/driver.rb:111:in `text'
/home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:301:in `block in send'
/home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:420:in `block in with_driver'
/home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:417:in `synchronize'
/home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:417:in `with_driver'
/home/kontena/kontena/kontena-websocket-client/lib/kontena/websocket/client.rb:300:in `send'
benchmark/websocket-benchmark.rb:64:in `block in websocket_benchark_sender'
benchmark/websocket-benchmark.rb:31:in `with_rate'
benchmark/websocket-benchmark.rb:63:in `websocket_benchark_sender'
benchmark/websocket-benchmark.rb:110:in `block (2 levels) in websocket_benchmark'
The reason for this happening is simple: the send thread calling Kontena::Websocket::Client#send will block on the Kontena::Websocket::Client::Connection#wait_socket_writable! with the @driver mutex held. This blocks the read thread calling Kontena::Websocket::Client#read -> #websocket_read from processing the data read from the socket (with_driver => driver.parse(...)), and the client deadlocks, unable to continue reading or writing:
I, [2017-07-25T15:12:48.638810 #13444] INFO -- Kontena::Websocket::Client: read_yield -> dequeue 4
I, [2017-07-25T15:12:48.738506 #13444] INFO -- Kontena::Websocket::Client: read_yield -> websocket_read...
I, [2017-07-25T15:12:48.838627 #13444] INFO -- Kontena::Websocket::Client::Connection: read size=4096 timeout=nil...
I, [2017-07-25T15:12:48.916796 #13444] INFO -- Kontena::Websocket::Client::Connection: wait write...
I, [2017-07-25T15:12:48.916900 #13444] INFO -- Kontena::Websocket::Client::Connection: read size=4096: #buf=4096
I, [2017-07-25T15:12:48.916947 #13444] INFO -- Kontena::Websocket::Client: read -> with_driver...
The client locking needs improvements to allow the socket.read -> driver.parse loop to continue if the driver.send -> socket.write would block, while also preventing any further driver.send -> socket.write calls from proceeding until the blocked write has proceeded in order to preserve write atomicity/ordering.
The text was updated successfully, but these errors were encountered:
When benchmarking (#13) against an echo server, the client will saturate at a point where it is writing messages faster than it is able to read them back from the server. When the client's socket read buffer starts to fill up, the echo server's socket write buffer will also fill up, and the echo server will stop reading from the socket. Once the client's socket write buffer also fills up, the client will end up in a deadlock situation where it fails with a write timeout:
The reason for this happening is simple: the send thread calling
Kontena::Websocket::Client#send
will block on theKontena::Websocket::Client::Connection#wait_socket_writable!
with the@driver
mutex held. This blocks the read thread callingKontena::Websocket::Client#read
->#websocket_read
from processing the data read from the socket (with_driver => driver.parse(...)
), and the client deadlocks, unable to continue reading or writing:The client locking needs improvements to allow the
socket.read
->driver.parse
loop to continue if thedriver.send
->socket.write
would block, while also preventing any furtherdriver.send
->socket.write
calls from proceeding until the blocked write has proceeded in order to preserve write atomicity/ordering.The text was updated successfully, but these errors were encountered: