potential client bug after 0.3.4 when sending slightly larger first message

[blbradley]

Hello! Thanks for the library.

I have been using this library to interface with the GDAX websocket feed. While upgrading to their new websocket protocol, the size of the subscription message increased. Now, I keep getting some UTF-8 error on log level DEBUG with 0.4.2 and 0.3.5:

2017-09-11 22:18:51,240 - ws4py - DEBUG - Error message received (1007) 'b'Invalid UTF-8 bytes''
2017-09-11 22:18:51,241 - root - WARNING - Closed down, code 1006: Going away

Under 0.4.1 and 0.4.0, I don't get the UTF-8 error, but the websocket closes in the same way:

2017-09-11 22:20:24,368 - root - WARNING - Closed down, code 1006: Going away

On 0.3.4, I don't get any errors.

Old subscription message: msg = """{"type": "subscribe","product_ids":["BTC-USD"]}"""
New subscription message: msg = """{"type": "subscribe", "product_ids": ["BTC-USD"], "channels": ["ticker", "level2"]}"""

Let me know if you need anything else.

[jmichiel]

I'm having a similar issue with 0.4.3 trying to use the Bitfinex v2 API:
either this error
DEBUG:ws4py:Error message received (1002) 'b'''
or this one
DEBUG:ws4py:Error message received (1007) 'b'Invalid UTF-8 bytes''

It happens almost every time you try to start multiple subscriptions on the same websocket.
Reverting to ws4py v0.3.4 works perfectly.

[jmichiel]

Apparently, it gets broken in v0.3.5 onwards: v0.3.4 works, v0.3.5 and later don't

[jmichiel]

I'm still looking into this and I see weird stuff happening after commit https://github.com/Lawouach/WebSocket-for-Python/search?q=75b88bdecd988694b0429c46cf5c3d6c719cc1f2&type=Commits&utf8=%E2%9C%93
which mainly introduces the _get_from_pending method in websocket.py

[jmichiel]

I think I know what's happening: as soon as more than 1 message is in the queue, the first message is decoded, but the rest of the buffer gets thrown away. If you're "lucky" one or more entire messages are thrown away, but the socket keeps working. As soon as only part of a message is thrown away, the parser will start halfway a frame and will bail out, closing the socket.
I'm looking into fixing it myself, but I'm afraid I don't know the code enough to really understand what's to be changed.
The main problem seems to be that there is no mechanism for the parsers to signal that they have not consumed all bytes. I was thinking of yielding negative numbers in this case, but I can't seem to get it to work...

[jmichiel]

Damn, if you change the code (see pull request) in opened() to

        self.send(json.dumps({'event': u'subscribe', 'channel': u'candles', 'key':'trade:5m:tBTCUSD'}))
        self.send(json.dumps({'event': u'subscribe', 'channel': u'candles', 'key':'trade:5m:tBCHUSD'}))
        self.send(json.dumps({'event': u'subscribe', 'channel': u'candles', 'key':'trade:5m:tBTGUSD'}))

You still get 'Invalid UTF-8 bytes' errors...

[hyperair]

This should be fixed in master now -- #239 has been merged, and the test I made for this issue in #220 passes. Additionally, this issue might be a dupe of #218.