AssertionError indexing snapshot not registered in DUOS

[dsotirho-ucsc]

Indexing error on anvildev due to regression from the merge of PR #6872

CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-indexer-anvildev-contribute_retry
start-time: 2025-02-05T17:14:14.699Z
end-time: 2025-02-05T17:23:11.088Z
query-string:

fields @timestamp, @message
| filter @requestId = 'bc916a15-8e8c-500e-899b-0a08111b5048'
| sort @timestamp asc
| limit 200

---

@timestamp	@message
2025-02-05 17:18:28.113	START RequestId: bc916a15-8e8c-500e-899b-0a08111b5048 Version: $LATEST
2025-02-05 17:18:28.113	[INFO] 2025-02-05T17:18:28.113Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.indexer.index_controller Worker handling message {'action': 'add', 'notification': {'transaction_id': '4b3447c7-0826-4fb3-8d8c-445b9c297da7', 'bundle_fqid': {'uuid': '677dd55c-3fa3-ab07-8c98-985d94d7577e', 'version': '2022-06-01T00:00:00.000000Z', 'source': {'id': '790795c4-49b1-4ac8-a060-207b92ea08c5', 'spec': 'tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:/0'}, 'table_name': 'anvil_dataset', 'batch_prefix': None}}, 'catalog': 'anvil'}, attempt #'1' (approx).
2025-02-05 17:18:28.114	[INFO] 2025-02-05T17:18:28.114Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.plugins.repository.tdr_anvil Bundle '677dd55c-3fa3-ab07-8c98-985d94d7577e' is a DUOS bundle
2025-02-05 17:18:28.128	[INFO] 2025-02-05T17:18:28.128Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.deployment Allocated new Boto3 client for 'secretsmanager' with ID 139621965464160
2025-02-05 17:18:28.298	[DEBUG] 2025-02-05T17:18:28.298Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.terra Query (296 characters total): '\n SELECT consent_group, data_modality, data_use_permission, datarepo_row_id, dataset_id, owner, principal_investigator, registered_identifier, source_datarepo_row_ids, title\n FROM datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset\n '
2025-02-05 17:18:29.442	[DEBUG] 2025-02-05T17:18:29.442Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.terra Job info: {"job_id": "8a19fc25-da19-408e-b038-80eaf2e15598", "total_rows": 1, "stats": {"totalBytesProcessed": "0", "totalBytesBilled": "0", "cacheHit": true, "searchStatistics": {"indexUsageMode": "UNUSED", "indexUnusedReasons": [{"code": "QUERY_CACHE_HIT", "message": "Search indexes are not used because the query was cached."}]}}, "query": "\n SELECT consent_group, data_modality, data_use_permission, datarepo_row_id, dataset_id, owner, principal_investigator, registered_identifier, source_datarepo_row_ids, title\n FROM datarepo-dev-e53e74aa.ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732.anvil_dataset\n "}
2025-02-05 17:18:29.622	[INFO] 2025-02-05T17:18:29.622Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.terra Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots/790795c4-49b1-4ac8-a060-207b92ea08c5'
2025-02-05 17:18:29.622	[DEBUG] 2025-02-05T17:18:29.622Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.terra … without request body
2025-02-05 17:18:30.180	[INFO] 2025-02-05T17:18:30.180Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.terra Got 200 response after 0.558s from GET to https://jade.datarepo-dev.broadinstitute.org/api/repository/v1/snapshots/790795c4-49b1-4ac8-a060-207b92ea08c5
2025-02-05 17:18:30.180	[DEBUG] 2025-02-05T17:18:30.180Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.terra … with response headers HTTPHeaderDict({'Date': 'Wed, 05 Feb 2025 17:18:30 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'eObjGrzz', 'Content-Type': 'application/json', 'Content-Length': '21219', 'Vary': 'Accept-Encoding,Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2025-02-05 17:18:30.180	[DEBUG] 2025-02-05T17:18:30.180Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.terra … with response body b'{"id":"790795c4-49b1-4ac8-a060-207b92ea08c5","name":"ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732","description":"Full vie...'
2025-02-05 17:18:30.183	[WARNING] 2025-02-05T17:18:30.181Z bc916a15-8e8c-500e-899b-0a08111b5048 azul.indexer.index_controller Worker failed to handle message {'action': 'add', 'notification': {'transaction_id': '4b3447c7-0826-4fb3-8d8c-445b9c297da7', 'bundle_fqid': {'uuid': '677dd55c-3fa3-ab07-8c98-985d94d7577e', 'version': '2022-06-01T00:00:00.000000Z', 'source': {'id': '790795c4-49b1-4ac8-a060-207b92ea08c5', 'spec': 'tdr:bigquery:gcp:datarepo-dev-e53e74aa:ANVIL_1000G_2019_Dev_20230609_ANV5_202306121732:/0'}, 'table_name': 'anvil_dataset', 'batch_prefix': None}}, 'catalog': 'anvil'}. Traceback (most recent call last): File "/var/task/azul/indexer/index_controller.py", line 173, in contribute contributions, replicas = self.transform(catalog, notification, delete) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/indexer/index_controller.py", line 225, in transform bundle = service.fetch_bundle(catalog, bundle_fqid) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/indexer/index_service.py", line 201, in fetch_bundle return plugin.fetch_bundle(bundle_fqid) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/plugins/repository/tdr.py", line 196, in fetch_bundle bundle = self._emulate_bundle(bundle_fqid) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/plugins/repository/tdr_anvil/init.py", line 333, in _emulate_bundle return self._duos_bundle(bundle_fqid) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/plugins/repository/tdr_anvil/init.py", line 494, in _duos_bundle duos_id, duos_info = self.tdr.get_duos(bundle_fqid.source) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/terra.py", line 666, in get_duos duos_id = json_str(json_dict(body['duosFirecloudGroup'])['duosId']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/types.py", line 75, in json_dict assert isinstance(v, dict), type(v) ^^^^^^^^^^^^^^^^^^^ AssertionError: <class 'NoneType'>
2025-02-05 17:18:30.189	END RequestId: bc916a15-8e8c-500e-899b-0a08111b5048
2025-02-05 17:18:30.189	REPORT RequestId: bc916a15-8e8c-500e-899b-0a08111b5048 Duration: 2076.56 ms Billed Duration: 2077 ms Memory Size: 4096 MB Max Memory Used: 149 MB Init Duration: 2588.26 ms

---

Formatted traceback

Traceback (most recent call last):
  File "/var/task/azul/indexer/index_controller.py", line 173, in contribute
    contributions, replicas = self.transform(catalog, notification, delete)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/azul/indexer/index_controller.py", line 225, in transform
    bundle = service.fetch_bundle(catalog, bundle_fqid)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/azul/indexer/index_service.py", line 201, in fetch_bundle
    return plugin.fetch_bundle(bundle_fqid)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/azul/plugins/repository/tdr.py", line 196, in fetch_bundle
    bundle = self._emulate_bundle(bundle_fqid)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/azul/plugins/repository/tdr_anvil/__init__.py", line 333, in _emulate_bundle
    return self._duos_bundle(bundle_fqid)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/azul/plugins/repository/tdr_anvil/__init__.py", line 494, in _duos_bundle
    duos_id, duos_info = self.tdr.get_duos(bundle_fqid.source)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/azul/terra.py", line 666, in get_duos
    duos_id = json_str(json_dict(body['duosFirecloudGroup'])['duosId'])
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/azul/types.py", line 75, in json_dict
    assert isinstance(v, dict), type(v)
           ^^^^^^^^^^^^^^^^^^^
AssertionError: <class 'NoneType'>