We should support retries

[sigurdm]

We should (optionally) retry queries that fail with HTTP 5xx and 429 response codes using truncated exponential backoff.

https://cloud.google.com/storage/docs/exponential-backoff outlines how it should be implemented

[jonasfj]

Specifically for 500 and 503:
https://cloud.google.com/storage/docs/xml-api/reference-status#500internal-server-error

Probably not for 429... I think that's only likely to make things worse..

[jonasfj]

We can probably just wrap the http.Client return from authClientService with package:http_retry.

As long as we only retry:

• 429 (maybe with a slightly longer delay)
• 504
• 500
• 503

See:

• https://cloud.google.com/storage/docs/xml-api/reference-status
• https://cloud.google.com/pubsub/docs/reference/error-codes
• https://cloud.google.com/datastore/docs/concepts/errors#error_codes