Request agent crashes when unable to contact RabbitMQ

[retzkek]

When RabbitMQ becomes unreachable the request agent crashes repeatedly, until systemd gives up on trying to restart it. It should have graceful handling of connection interruptions and retry logic.

Oct 18 15:22:49 gratiav2-1 systemd[1]: graccreq.service holdoff time over, scheduling restart.
Oct 18 15:22:49 gratiav2-1 systemd[1]: Started GRACC Request Agent.
Oct 18 15:22:49 gratiav2-1 systemd[1]: Starting GRACC Request Agent...
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: DEBUG:pika.adapters.select_connection:Using EPollPoller
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: DEBUG:pika.callback:Added: {'callback': <bound method _CallbackResult.set_value_once of <pika.adapters.blocking_connection._CallbackResult object at 0x2b5f0e0>>, 'only': None, 'one_s
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: DEBUG:pika.callback:Added: {'callback': <bound method _CallbackResult.set_value_once of <pika.adapters.blocking_connection._CallbackResult object at 0x2b5f098>>, 'only': None, 'one_s
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: DEBUG:pika.callback:Added: {'callback': <bound method _CallbackResult.set_value_once of <pika.adapters.blocking_connection._CallbackResult object at 0x2b5f128>>, 'only': None, 'one_s
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: DEBUG:pika.callback:Added: {'callback': <bound method SelectConnection._on_connection_start of <pika.adapters.select_connection.SelectConnection object at 0x2b5cd10>>, 'only': None, 
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: INFO:pika.adapters.base_connection:Connecting to 129.79.53.45:5672
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: WARNING:pika.adapters.base_connection:Connection to 129.79.53.45:5672 failed: [Errno 111] Connection refused
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: INFO:pika.adapters.base_connection:Connecting to 2001:18e8:2:6::10e:5672
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: WARNING:pika.adapters.base_connection:Connection to 2001:18e8:2:6::10e:5672 failed: [Errno 101] Network is unreachable
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: WARNING:pika.connection:Could not connect, 0 attempts left
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: DEBUG:pika.callback:Processing 0:_on_connection_error
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: DEBUG:pika.callback:Calling <bound method _CallbackResult.set_value_once of <pika.adapters.blocking_connection._CallbackResult object at 0x2b5f0e0>> for "0:_on_connection_error"
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: Traceback (most recent call last):
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: File "/usr/bin/graccreq", line 9, in <module>
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: load_entry_point('graccreq==2.2', 'console_scripts', 'graccreq')()
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: File "/usr/lib/python2.7/site-packages/graccreq/OverMind.py", line 106, in main
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: overmind.run()
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: File "/usr/lib/python2.7/site-packages/graccreq/OverMind.py", line 37, in run
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: self.createConnection()
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: File "/usr/lib/python2.7/site-packages/graccreq/OverMind.py", line 56, in createConnection
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: self._conn = pika.adapters.blocking_connection.BlockingConnection(self.parameters)
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 339, in __init__
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: self._process_io_for_connection_setup()
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 374, in _process_io_for_connection_setup
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: self._open_error_result.is_ready)
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 395, in _flush_output
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: raise exceptions.ConnectionClosed()
Oct 18 15:22:49 gratiav2-1 graccreq[2753]: pika.exceptions.ConnectionClosed
Oct 18 15:22:49 gratiav2-1 systemd[1]: graccreq.service: main process exited, code=exited, status=1/FAILURE
Oct 18 15:22:49 gratiav2-1 systemd[1]: Unit graccreq.service entered failed state.
Oct 18 15:22:49 gratiav2-1 systemd[1]: graccreq.service failed.
Oct 18 15:22:50 gratiav2-1 systemd[1]: graccreq.service holdoff time over, scheduling restart.
Oct 18 15:22:50 gratiav2-1 systemd[1]: start request repeated too quickly for graccreq.service
Oct 18 15:22:50 gratiav2-1 systemd[1]: Failed to start GRACC Request Agent.
Oct 18 15:22:50 gratiav2-1 systemd[1]: Unit graccreq.service entered failed state.
Oct 18 15:22:50 gratiav2-1 systemd[1]: graccreq.service failed.