monitor/ensure connection health for RabbitMQ message consumption

[jmartin-sul]

NOTE: the solution for this will likely look very similar to the solution for sul-dlss/dor-services-app#5190, so whoever picks this ticket up should probably either grab that other one also, or mark that one as blocked. Then we should try to use the same solution in both codebases for consistency.

Yesterday (see Slack thread) I noticed that H2 on QA and stage apparently lost its ability to consume messages from RabbitMQ, leading to loss of ability to detect successful item deposits. This caused the integration tests to fail, because they were stuck waiting on test objects that never left the "depositing" state in H2.

Going to the H2 VM and running the rabbitmq:setup rake task fixed things. Deposit completion was detected after that, and integration tests passed.

We suspect that this might've been an artifact of the wave of VM reboots that were necessary to restore functionality after a storage outage hit most of our infrastructure.

Two questions:

1. Is there a way to proactively monitor the connection health? For the opposite direction in pres cat, we have an okcomputer check that monitors RabbitMQ connection health. This situation is slightly different since we want to monitor the ability to receive messages, not to send them. But H2 is a web app, so whatever the check looks like, it could still be done via okcomputer, if we want.
2. Is there a way to proactively ensure connection health? Since we only seem to run into this once in a blue moon, possibly only after events like system reboots (we haven't tried to reproduce the issue yet), maybe this isn't worth the effort. One idea would be to e.g. run the rabbitmq:setup rake task on deploy, but the author of this ticket is unsure if that might have undesirable side effects. E.g. is that Rake task non-destructive? Would it possibly drop in flight messages if that Rake task were run while the connection was fine?