There is hang when use read function in thriftpy2/transport/socket.py

[leeyh0216]

Hello, I think there might be some bug on socket.py

Environment

• thriftpy2 == 0.4.14
• impyla == 0.16.3
• hive server == 2.3.2

Problem

When i query large data on hive2 server with impyla library, query is going hang.
I got out from hang via ctrl + c and i see stack trace.

File ".../bug_env/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 198, in _trans_read_all
    return read_all(sz)
  File ".../bug_env/lib/python3.6/site-packages/thriftpy2/transport/socket.py", line 110, in read
    buff = self.sock.recv(sz)

In socket.py, it uses sock.recv with sz(data length to receive).

buff = self.sock.recv(sz)

sock.recv function does not always receive data at once(sz). It's parameter(sz) is just max size of buffer.(see https://manpages.debian.org/buster/manpages-dev/recv.2.en.html)

So, I think read function have to change like this.

def read(self, sz):
        buf_ret = None
        byte_to_recv = sz
        while byte_to_recv > 0:
            try:
                buff = self.sock.recv(byte_to_recv)
                if buf_ret is None:
                    buf_ret = buff
                else:
                    buf_ret += buff
                    print("append to buff_ret")
                print("byte_to_recv len: {}, buff len: {}".format(byte_to_recv, len(buff)))
                byte_to_recv -= len(buff)
                print("recv: {}".format(byte_to_recv))
...
       return buf_ret

[Rosadosa]

Seems to be the same issue as we had?
#162

[leeyh0216]

@Rosadosa
Yes, I think it's stacktrace looks similiar to #162 issue's last stacktrace.
Can you try change source code in your virtual environment like above and test your code?
When it works, i think this issue same with issue #162

[ethe]

Thanks, I am looking for reproducing your case and testing your solution.

[courtipa]

@leeyh0216 thanks for your solution. I was facing the exact same problem. It's now solved :)

Environment:

• impyla == 0.16.3
• thriftpy2 == 0.4.14