Internal error in API server when trying export to Terra

[danxmoran]

Can be reproduced at https://broad-gdr-encode.appspot.com. Clicking the export button causes an alert to be raised saying "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application."

With gcloud --project=broad-gdr-encode app logs tail -s api, I found:

Traceback (most recent call last):    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app      response = self.full_dispatch_request()    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request      rv = self.handle_user_exception(e)    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception      reraise(exc_type, exc_value, tb)    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request      rv = self.dispatch_request()    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request      return self.view_functions[rule.endpoint](**req.view_args)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/decorator.py", line 66, in wrapper      response = function(request)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/validation.py", line 122, in wrapper      response = function(request)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/validation.py", line 293, in wrapper      return function(request)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/decorator.py", line 42, in wrapper      response = function(request)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/parameter.py", line 195, in wrapper      return function(**kwargs)    File "/app/data_explorer/controllers/export_url_controller.py", line 376, in export_url_post      query = _get_filter_query(filter_arr)    File "/app/data_explorer/controllers/export_url_controller.py", line 277, in _get_filter_query      for k, v in current_app.config['SAMPLE_FILE_COLUMNS'].iteritems()  AttributeError: 'list' object has no attribute 'iteritems'

The problem is a type mismatch here (it should default to a dict, not a list). I'll send a PR.

[danxmoran]

Fixing the above error (and a few other easy ones) led me to:

2018-11-01 15:20:27 api[20181101t110120]  Traceback (most recent call last):    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app      response = self.full_dispatch_request()    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request      rv = self.handle_user_exception(e)    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception      reraise(exc_type, exc_value, tb)    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request      rv = self.dispatch_request()    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request      return self.view_functions[rule.endpoint](**req.view_args)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/decorator.py", line 66, in wrapper      response = function(request)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/validation.py", line 122, in wrapper      response = function(request)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/validation.py", line 293, in wrapper      return function(request)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/decorator.py", line 42, in wrapper      response = function(request)    File "/usr/local/lib/python2.7/site-packages/connexion/decorators/parameter.py", line 195, in wrapper      return function(**kwargs)    File "/app/data_explorer/controllers/export_url_controller.py", line 387, in export_url_post      gcs_path = _write_gcs_file(entities)    File "/app/data_explorer/controllers/export_url_controller.py", line 174, in _write_gcs_file      copied_samples_blob.rewrite(samples_blob)    File "/usr/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 1428, in rewrite      source._encryption_key, source=True))  AttributeError: 'NoneType' object has no attribute '_encryption_key'

It looks like this call is returning None. Is samples_blob meant to be the file written by the indexer?

[melissachang]

Yes, added in DataBiosphere/data-explorer-indexers#81

Please reindex

[melissachang]

(To clarify, please sync indexer to head and reindex)

Let me know if that works. I assume #204 isn't necessary.

[danxmoran]

#204 does fix other bugs in the API layer

[melissachang]

I'm confused, what is the state of events after deleting the index, running the indexer at master, and running this repo at master?

What is the exact bug that #204 fixes? Can this bug be reproduced at http://test-data-explorer.appspot.com, or running this repo at master?

[danxmoran]

#204 fixes the bugs I hit before this error:

1. A type error where a field is initialized to a list, then optionally changed to be a dictionary, then assumed to be a dictionary everywhere else
2. A "key not found" error when EXTRA_FACET_INFO isn't initialized
3. A "key not found" error when the sample ID column is something other than "sample_id"

I am running the repo at master. They aren't reproducible at the test explorer because the hard-coded bits of the API align with the dataset / config of that explorer.

[danxmoran]

Also can confirm that the branch we used for indexing (DataBiosphere/data-explorer-indexers#92) included DataBiosphere/data-explorer-indexers#81, and a samples export file was written to gs://broad-gdr-encode-export-samples/samples. It looks like we need to merge in DataBiosphere/data-explorer-indexers#87, which changed the naming convention on the indexer side.

For now I'll just rename the existing export object.

[danxmoran]

Moving the exported file to gs://broad-gdr-encode-export-samples/encode-None-samples got the export through the final error (is the middle piece expected to be None?)

[bfcrampton]

Thanks for fixing those things @danxmoran, just approved the PR with one note about EXTRA_FACET_INFO.

The None in the middle should be your os username. This was added so that multiple people indexing the same dataset don't overwrite each other and the test-data-explorer.