Sage data node returns 504 error

[tschaffter]

@ymnliu made a submission today to the date annotation task and received a workflow error file. Below is the main issue. The error seems to happen when the the controller tries to create an annotation store in the data node. This is a new issue that likely results from adding an nginx container in front of the data node API container.

Additional information:

• The nginx container receives requests on the port 80, which is the port that the data node API used to receive request.
• The nginx and API containers are both running. The Swagger UI of the API container is accessible in the browser via nginx.
• I don't remember whether I ever had a successful submission since I added nginx in front of the data node. I just sent a submission.

@thomasyu888 Do you have an idea what the issue may be?

Next:

• I just made a new submission and will update this ticket with the result.

STDERR: 2021-11-23T17:03:14.947813370Z 	Created Annotation Store
STDERR: 2021-11-23T17:03:14.947816210Z 	Traceback (most recent call last):
STDERR: 2021-11-23T17:03:14.947819089Z 	  File "/usr/local/bin/nlp-cli", line 8, in <module>
STDERR: 2021-11-23T17:03:14.947822156Z 	    sys.exit(main())
STDERR: 2021-11-23T17:03:14.947824865Z 	  File "/usr/local/lib/python3.8/site-packages/nlpsandboxclient/cli/__main__.py", line 19, in main
STDERR: 2021-11-23T17:03:14.947827835Z 	    cli()
STDERR: 2021-11-23T17:03:14.947830544Z 	  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
STDERR: 2021-11-23T17:03:14.947833953Z 	    return self.main(*args, **kwargs)
STDERR: 2021-11-23T17:03:14.947836740Z 	  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1062, in main
STDERR: 2021-11-23T17:03:14.947839618Z 	    rv = self.invoke(ctx)
STDERR: 2021-11-23T17:03:14.947842361Z 	  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
STDERR: 2021-11-23T17:03:14.947845607Z 	    return _process_result(sub_ctx.command.invoke(sub_ctx))
STDERR: 2021-11-23T17:03:14.947849649Z 	  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
STDERR: 2021-11-23T17:03:14.947853260Z 	    return _process_result(sub_ctx.command.invoke(sub_ctx))
STDERR: 2021-11-23T17:03:14.947856066Z 	  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
STDERR: 2021-11-23T17:03:14.947858910Z 	    return ctx.invoke(self.callback, **ctx.params)
STDERR: 2021-11-23T17:03:14.947861731Z 	  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
STDERR: 2021-11-23T17:03:14.947864573Z 	    return __callback(*args, **kwargs)
STDERR: 2021-11-23T17:03:14.947867324Z 	  File "/usr/local/lib/python3.8/site-packages/nlpsandboxclient/cli/datanode_cli.py", line 50, in store_annotations
STDERR: 2021-11-23T17:03:14.947870301Z 	    client.store_annotations(
STDERR: 2021-11-23T17:03:14.947873716Z 	  File "/usr/local/lib/python3.8/site-packages/nlpsandboxclient/client.py", line 645, in store_annotations
STDERR: 2021-11-23T17:03:14.947876711Z 	    _store_annotation(
STDERR: 2021-11-23T17:03:14.947879523Z 	  File "/usr/local/lib/python3.8/site-packages/nlpsandboxclient/client.py", line 205, in _store_annotation
STDERR: 2021-11-23T17:03:14.947882525Z 	    return annotation_obj.get()
STDERR: 2021-11-23T17:03:14.947885310Z 	  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 771, in get
STDERR: 2021-11-23T17:03:14.947888328Z 	    raise self._value
STDERR: 2021-11-23T17:03:14.947890995Z 	  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker
STDERR: 2021-11-23T17:03:14.947894125Z 	    result = (True, func(*args, **kwds))
STDERR: 2021-11-23T17:03:14.947896895Z 	  File "/usr/local/lib/python3.8/site-packages/nlpsandbox/api_client.py", line 203, in __call_api
STDERR: 2021-11-23T17:03:14.947899831Z 	    raise e
STDERR: 2021-11-23T17:03:14.947905249Z 	  File "/usr/local/lib/python3.8/site-packages/nlpsandbox/api_client.py", line 196, in __call_api
STDERR: 2021-11-23T17:03:14.947908226Z 	    response_data = self.request(
STDERR: 2021-11-23T17:03:14.947910955Z 	  File "/usr/local/lib/python3.8/site-packages/nlpsandbox/api_client.py", line 455, in request
STDERR: 2021-11-23T17:03:14.947913838Z 	    return self.rest_client.POST(url,
STDERR: 2021-11-23T17:03:14.947916575Z 	  File "/usr/local/lib/python3.8/site-packages/nlpsandbox/rest.py", line 271, in POST
STDERR: 2021-11-23T17:03:14.947919435Z 	    return self.request("POST", url,
STDERR: 2021-11-23T17:03:14.947922275Z 	  File "/usr/local/lib/python3.8/site-packages/nlpsandbox/rest.py", line 228, in request
STDERR: 2021-11-23T17:03:14.947925151Z 	    raise ServiceException(http_resp=r)
STDERR: 2021-11-23T17:03:14.947927864Z 	nlpsandbox.exceptions.ServiceException: (504)
STDERR: 2021-11-23T17:03:14.947930591Z 	Reason: Gateway Time-out
STDERR: 2021-11-23T17:03:14.947933289Z 	HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.19.6', 'Date': 'Tue, 23 Nov 2021 17:03:14 GMT', 'Content-Type': 'text/html', 'Content-Length': '167', 'Connection': 'keep-alive'})
STDERR: 2021-11-23T17:03:14.947936528Z 	HTTP response body: <html>
STDERR: 2021-11-23T17:03:14.947939806Z 	<head><title>504 Gateway Time-out</title></head>
STDERR: 2021-11-23T17:03:14.947943081Z 	<body>
STDERR: 2021-11-23T17:03:14.947945815Z 	<center><h1>504 Gateway Time-out</h1></center>
STDERR: 2021-11-23T17:03:14.947948919Z 	<hr><center>nginx/1.19.6</center>
STDERR: 2021-11-23T17:03:14.947951895Z 	</body>
STDERR: 2021-11-23T17:03:14.947954667Z 	</html>

[thomasyu888]

Will look at it when I'm back from vacation next Monday.

[thomasyu888]

I googled real quick and found this: https://easycloudsupport.zendesk.com/hc/en-us/articles/360002057472-How-to-Fix-504-Gateway-Timeout-using-Nginx

Your submission went through which means that it is probably just another error code we have to potentially account for.

[tschaffter]

Referencing the log file: https://www.synapse.org/#!Synapse:syn26477061

Will look at it when I'm back from vacation next Monday.

You shouldn't even have sent this message. But I will not complain too much. Thanks @thomasyu888 !

[tschaffter]

The potential issue described below is not related to this issue, however it shows that the controller was able to create an annotation store for a submission created via nginx and after Sijia shared his issue. Therefore it's possible that we are facing a random issue. I will look at the link Tom shared and further investigate.

Note: I haven't seen the RandomDisconnected issue since nginx has been added in front of the data node. More submissions need to be made to confirm this observation.

Incorrect annotation store name for Mayo Clinic dataset?

The controller successfully created an annotation store for the submission I made after opening this ticket, for the i2b2 dataset. I confirmed this using its Swagger UI.

I can't find a similar entry for the Mayo Clinic dataset. However, it seems that an annotation store for this dataset has been created after I made my submission:

• my submission ID is 9717406 (as given by the submission dashboard). It has an annotation store for the i2b2 dataset (see above) but not for the Mayo clinic dataset despite having been successfully evaluated on it.
• there is another submission with the ID 9717408 made by X which has an annotation store for the Mayo Clinic dataset (see below) but not for the i2b2 dataset (see above).

Given how close the ID 9717406 and 9717408 are, isn't the second an internal submission ID for the mayo clinic dataset? If yes, we should prefer to use the main ID to name the annotation stores for Mayo Clinic dataset. Otherwise, it will be difficult to retrieve the prediction of a given submission on multiple datasets.