Gracefully handling and retrying on certain Thrift transport exceptions

[konklone]

I've been trying out the Thrift transport backend for rubberband. When the underlying ElasticSearch instance is restarted, the next search request by the client yields a Thrift::TransportException::UNKNOWN_METHOD exception with the message "closed stream", which rubberband catches and transmutes into a ElasticSearch::ConnectionFailed exception. The second request, and all future requests, yields an IOError with the message "Broken pipe".

The rubberband source has the comment:

#TODO Thrift::ApplicationException, Thrift::ProtocolException, IOError.. retryable or fatal?

I was able to fix both the Thrift::TransportException::UNKNOWN_METHOD and the IOError case by rescuing them and generating a new ElasticSearch client and replacing the one I'd been using - after doing so, the next and future requests succeeded.

It would be nice to build this forgiveness into the rubberband client itself, since ElasticSearch restarts should be painless and invisible to clients.

[grantr]

Sorry must have missed this issue!

Seems like this could be fixed by making the thrift errors retryable, then the same options and code used to retry http transport errors would work for thrift. It would be nice to have a list of thrift errors that are likely to be transient.