Possible bug involving HTTPS chunk encoding

[clbarnes]

catpy has started throwing errors on some large requests - digging into them, it seems to be related to a new(ish) openssl client version on linux, which the openssl developers claim is indicative of a server-side issue, even though the old version of openssl does not trigger it.

Traceback (most recent call last):                                                                                                                        
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 685, in _update_chunk_length                
    self.chunk_left = int(line, 16)                                                                                                                       
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 425, in _error_catcher
    yield
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 752, in read_chunked
    self._update_chunk_length()
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 689, in _update_chunk_length
    raise httplib.IncompleteRead(line) 
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 560, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 781, in read_chunked
    self._original_response.close()
  File "/home/barnesc/.pyenv/versions/3.7.5/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 443, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))


During handling of the above exception, another exception occurred:           

Traceback (most recent call last):     
  File "/home/barnesc/.pyenv/versions/brain/bin/ascending", line 11, in <module>                                                                             
    load_entry_point('brain-completion', 'console_scripts', 'ascending')()    
  File "/home/barnesc/work/code/brain_completion/brain_completion/scripts/section_a/ascending.py", line 148, in main                                         
    catmaid.delete_annotations(creates_annotations)                           
  File "/home/barnesc/work/code/brain_completion/brain_completion/catmaid_interface.py", line 393, in delete_annotations                                     
    return self.remove_annotations(annotations, entity_ids)                   
  File "/home/barnesc/work/code/brain_completion/brain_completion/catmaid_interface.py", line 389, in remove_annotations                                     
    return self.post(("annotations", "remove"), data)                         
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/catpy/client.py", line 66, in post                                        
    return self.fetch(relative_url, method="POST", data=data, raw=raw, **kwargs)                                                                             
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/catpy/applications/base.py", line 34, in fetch                            
    return self._catmaid.fetch(*args, **kwargs)                               
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/catpy/client.py", line 230, in fetch                                      
    response = self._session.post(url, data=data, **kwargs)                   
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/sessions.py", line 581, in post                                  
    return self.request('POST', url, data=data, json=json, **kwargs)          
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/sessions.py", line 533, in request                               
    resp = self.send(prep, **send_kwargs)                                     
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/sessions.py", line 686, in send                                  
    r.content                          
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/models.py", line 828, in content                                 
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''    
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/models.py", line 753, in generate                                
    raise ChunkedEncodingError(e)      
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) 

Issue chain:

• catpy HTTPS chunk encoding issue? catpy#43
• requests Error decoding chunked response in _update_chunk_length: ValueError: invalid literal for int() with base 16: b'HTTP/1.1 200 OK\r\n' psf/requests#4248
• urllib3 Getting exception in _update_chunk_length urllib3/urllib3#1516
• openssl malformed HTTPS chunked output in 1.1.1c, but working in 1.0.2s in Linux openssl/openssl#9360

Our chunking is, of course, handled by Django, so maybe we need to add another link to the chain...