MissingIdentityError on token refresh

[csparker247]

Last week, I transitioned to using the experimental UserApp class with JSONTokenStorage:

from pathlib import Path
from globus_sdk.experimental.globus_app import UserApp, GlobusAppConfig
from globus_sdk.experimental.tokenstorage import JSONTokenStorage
import globus_sdk

# set up app
app_name = 'foo'
app_ns = 'com.foo'
client_id = '#####'
token_store_path =  Path().home() / '.globusconf.json'
token_store = JSONTokenStorage(token_store_path, namespace=app_ns)
app = UserApp(app_name=app_name, client_id=client_id, 
              config=GlobusAppConfig(token_storage=token_store, request_refresh_tokens=True))

# login
app.login()

# create transfer client
client = globus_sdk.TransferClient(app=app, app_name=app_name)

# test endpoint access
# note: I still have to do this to make sure I'm completely logged in
endpoint_uuid = '#####'
try:
  client.operation_ls(endpoint_uuid, path='/')
except globus_sdk.TransferAPIError as err:
  if not err.info.consent_required:
    raise
  client.add_app_scope(err.info.consent_required.required_scopes)
  app.login()  # login again now that scopes have been added

### code to do some file transfers ###

This was working pretty consistently in my tests, so I set up some script runs on timers to do some transfers. The first of these timed script executions completed successfully, but after a while, everything started failing with a MissingIdentityError at the point that I test the login with operation_lc:

Traceback (most recent call last):
  File "/home/FOO/transfer/copy_tool/copy_tool.py", line 93, in <module>
    main()
  File "/home/FOO/transfer/copy_tool/copy_tool.py", line 40, in main
    client.operation_ls(uuid, path="/")
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/services/transfer/client.py", line 1214, in operation_ls
    self.get(f"operation/endpoint/{endpoint_id}/ls", query_params=query_params)
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/client.py", line 254, in get
    return self.request(
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/client.py", line 424, in request
    r = self.transport.request(
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/transport/requests.py", line 315, in request
    self._set_authz_header(authorizer, req)
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/transport/requests.py", line 251, in _set_authz_header
    authz_header = authorizer.get_authorization_header()
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/authorizers/renewing.py", line 168, in get_authorization_header
    self.ensure_valid_token()
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/authorizers/renewing.py", line 162, in ensure_valid_token
    self._get_new_access_token()
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/authorizers/renewing.py", line 135, in _get_new_access_token
    self.on_refresh(res)
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/experimental/tokenstorage/base.py", line 108, in store_token_response
    self.store_token_data_by_resource_server(token_data_by_resource_server)
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/experimental/globus_app/_validating_token_storage.py", line 121, in store_token_data_by_resource_server
    self._validate_token_data_by_resource_server_meets_identity_requirements(
  File "/home/FOO/transfer/venv/lib/python3.10/site-packages/globus_sdk/experimental/globus_app/_validating_token_storage.py", line 194, in _validate_token_data_by_resource_server_meets_identity_requirements
    raise MissingIdentityError(
globus_sdk.experimental.globus_app.errors.MissingIdentityError: Token grant response doesn't contain an id_token. This normally occurs if the auth flow didn't include 'openid' alongside other scopes.

After debugging this a bit, it seems that the token has expired and the app is trying to refresh the token, but the new token for the transfer.api.globus.org resource server does not include the identity_id field. Despite the suggestion in the exception message, I cannot see how to add the openid scope to the transfer client scopes (my attempts to do so produce an UnmetScopeRequirementsError).

The only thing I can do to get this working again is to delete my token store .json file. app.login(force=True) does force me to login again, but does not fix the MissingIdentityError. I do understand that I'm using an experimental API.

[sirosen]

Hi there! I think you've uncovered a bug in the implementation.

Every login flow needs to have the openid scope present, but it sounds like the logic which determines which scopes to present does not ensure this correctly. I'm still trying to work out exactly what's going wrong here -- I think it hinges on the scope requirement for openid being implicit rather than explicit -- so that we can fix it. Thanks for the bug report!

[sirosen]

One minor note on this, as we're working on it right now. I was incorrect about the cause -- there isn't a likely case in which the openid scope is missing. Instead, the issue is that there's a code path under a UserApp which expects to be getting tokens from a login flow, but is used generically to handle both new tokens from login and the renewed tokens from a refresh.
The token response from refresh has some different characteristics in terms of the data it contains, and needs explicit handling.

We have a series of changes to apply, which will make it simple for the two cases to be distinguished, and will add the appropriate handling for token refreshes.

[csparker247]

Just checking in and it seems like this will be fixed by #1055 in the next release, correct? For my own planning, is there an expected date for the next release?

[derek-globus]

Yep, this was fixed by #1055. We're expecting to cut a release this week (no specific date within the week as of yet) which includes this bugfix.

[csparker247]

Excellent! Thank you all for your help and the quick fix.

[derek-globus]

Of course, thanks for reaching out when you did!
You helped us catch a pretty gnarly bug that would've otherwise likely made it out of experimental!