calling list_accessible_customers with invalid refresh_token retries forever

[yuichiro12]

Hi, I called list_accessible_customers with invalid refresh_token:

# `client` is initialized with invalid refresh_token
client.service.customer.list_accessible_customers

and I got this GRPC error:

#<GRPC::Unavailable: 14:Getting metadata from plugin failed with error: #<Signet::AuthorizationError: Authorization failed.  Server message:
{
  "error": "invalid_grant",
  "error_description": "Token has been expired or revoked."
}>. debug_error_string:{"created":"@1604980241.901884600","description":"Getting metadata from plugin failed with error: #<Signet::AuthorizationError: Authorization failed.  Server message:\n{\n  "error": "invalid_grant",\n  "error_description": "Token has been expired or revoked."\n}>","file":"src/core/lib/security/credentials/plugin/plugin_credentials.cc","file_line":93,"grpc_status":14}>

Is this response supposed to be returned with code 14? I think code 16 (Unauthenticated) is desirable.
The problem is, GRPC response with code 14 are included in default retry_policy and it retries forever because of invalid refresh_token.

See retry_policy defaults here:
https://github.com/googleads/google-ads-ruby/blob/master/lib/google/ads/google_ads/v5/services/customer_service/client.rb#L67

Thanks.

[javoeria]

Hi, I'm using the last version of the gem and I have the same problem too.
This error occurs in all methods, not only in "list_accessible_customers". For example, I'm trying to get basic information of an account and I got the same "GRPC::Unavailable" error after I manually cancelled the operation because the method never stops.
Maybe @mcloonan can help us.
Thanks.

[mcloonan]

I'll look into updating our retry defaults to something that might make more sense. In the meanwhile if you want, you can actually override these yourself by calling "configure" on the created service. See an example here: https://github.com/googleads/google-ads-ruby/blob/master/examples/misc/set_custom_client_timeouts.rb

If you do mess around with this and find new defaults you like, let me know what they are so we can take that into account as we're looking into this.

[javoeria]

The problem here is what @yuichiro12 said, the "grpc_status" param of this error is 14 so it retries forever.
I found a temporary fix for me removing the code in the retry policy like in the example and it works:
config.rpcs.search.retry_policy = { retry_codes: [4] }

Also I think updating code to 16 (Unauthenticated: The request does not have valid authentication credentials for the operation.) is the better option.
Thanks

[mcloonan]

I think part of the problem is that the API is returning the wrong code for this case. Case 14 (unavailable) is usually going to be a temporary API outage, and something that can be retried forever. Case 16 (unauthenticated) is something that you as the developer have to fix, and so shouldn't be retried at all. So in theory the library is actually behaving correctly, since it sees error code 14. We're going to look into why the wrong error code is getting returned, and if we can figure that out the library should simply start behaving correctly.

In the meanwhile, we're going to look into updating how we handle error 14 (unavailable) so that there are a maximum number of retries, since getting an error message sooner will help people start to diagnose the problem, even if the error message is incorrect.

[viacheslav-rostovtsev]

This seems to be an intended GRPC behavior:

it 'should receive UNAVAILABLE if call credentials plugin fails'
https://github.com/grpc/grpc/blob/95ab12cf8fca2d92b4a097369dd8249cf8997b9f/src/ruby/spec/generic/client_stub_spec.rb#L239

The reason for this behavior:
"Typically these failures are symptomatic of a busy oauth2 auth server and calls should be retried in this case." ( grpc/grpc#13322 )

On whether it's a bug in gapic-generator-ruby

• ruby gapic generated libraries do not unwrap/rewrap the errors, so they surface the UNAVAILABLE they get from the GRPC.

On potential mitigation

• updating retry policy for google-ads-ruby provides mitigation for this (and makes sense in any case)

• we can try probing all UNAVAILABLE for containing Signet::AuthorizationError and converting them to UNATHENTICATED.

  • Signet seems to be returning different status codes when server experiencing issues versus when a genuine authorization error occurs:
    e.g. Signet::AuthorizationError on Auth errors, Signet::RemoteServerError when it gets 5xx from the server, Signet::UnexpectedStatusError for other issues
    https://github.com/googleapis/signet/blob/cd5d2792055e99e188b81a4210e717ddb4071fa3/spec/signet/oauth_2/client_spec.rb#L567

• more generically there could be a mechanism to tell a client library: """ treat these errors from my auth plugin as UNATHENTICATED when received inside UNAVAILABLE """

  • this looks too complicated

I'll try to figure out if this impacts Cloud libraries as well.

[ondrejbartas]

Any news about this issue?

[troex]

For now ended up with really quick and dirty workaround:

Google::Ads::GoogleAds::V10::Services::GoogleAdsService::Client.configure do |c|
  c.retry_policy = { initial_delay: 5.0, max_delay: 60.0, multiplier: 1.3, retry_codes: [4] }
end

Definetly looking for a better solution, but for noq it unleast doesn't lock background workers in infine loop

[mcloonan]

Here's a link to the issue tracker for the underlying issue in the appropriate repo: googleapis/google-auth-library-ruby#366

If you're affected, consider adding a comment there so they can keep track of how widespread this issue is.

[diegosaul402]

This issue is still happening to me

[mcloonan]

I urge anyone affected by this to comment on the issue in google-auth-library-ruby (linked above) so that the team responsible for that layer prioritizes this fix. It's not something that our library can control, unfortunately.

[tmck-code]

This was also happening to me, ended up solving by using the ruby Timeout lib

begin
  Timeout::timeout(10) do
    client.service.customer.list_accessible_customers.resource_names
  end
rescue Timeout::Error
  # rescue logic happens here, if list_accessible_customers took >10 seconds
end

[jheathco]

Is there a way to determine what the actual error was if the timeout exception is triggered?

This was also happening to me, ended up solving by using the ruby Timeout lib

begin
  Timeout::timeout(10) do
    client.service.customer.list_accessible_customers.resource_names
  end
rescue Timeout::Error
  # rescue logic happens here, if list_accessible_customers took >10 seconds
end