Troubleshooting Paul's submission

[tschaffter]

@paulheider forked this date annotator example, manually built the docker image before pushing it to Synapse. He then submitted it and received the following error that he shared with me by email:

Hello Paul-M-Heider, Your submission (id: 9719900) is invalid, below are the invalid reasons: API api/v1/tool endpoint not implemented or implemented incorrectly. Make sure correct tool object is returned. .../api/v1/ui not
 implemented or implemented incorrectly. Sincerely, Challenge Administrator

@paulheider I successfully performed the above operation and received the scores for the i2b2 and Mayo Clinic dataset (MCW data site may be down, we are investigating).

I have a few questions:

• Did you follow these instructions to build and push the image to Synapse?
• Did you submit the Docker image to the queue NLP sandbox - Date Annotator?

A quick look at your fork reveal that you have not yet customized the tool info returned by the endpoint /api/v1/tool. This is good for now as we should change as little as possible before we figure out what the current issue is.

[tschaffter]

Here are the emails I received for the submission I made:

Hello NLP Sandbox Bots,

Your submission (id: 9719977) is scored, below are your results for i2b2-phi-20210622:

location_F1_instance_relax : 0.25
location_precision_instance_relax : 0.26
location_recall_instance_relax : 0.25
location_F1_instance_strict : 0.25
location_precision_instance_strict : 0.26
location_recall_instance_strict : 0.25
location_F1_token_strict : 0.34
location_precision_token_strict : 0.37
location_recall_token_strict : 0.31

Sincerely,
Challenge Administrator

Hello NLP Sandbox Bots,

Your submission (id: 9719977) is scored, below are your results for mayo-clinic-20211026:

location_F1_instance_relax : 0.3
location_precision_instance_relax : 0.25
location_recall_instance_relax : 0.39
location_F1_instance_strict : 0.3
location_precision_instance_strict : 0.25
location_recall_instance_strict : 0.39
location_F1_token_strict : 0.3
location_precision_token_strict : 0.25
location_recall_token_strict : 0.39

Sincerely,
Challenge Administrator 

[paulheider]

1. Did you follow these instructions to build and push the image to Synapse?

Yes. I followed the above instructions with two caveats.

• The docker-compose.yml sets the default name for the docker image to something looking a lot more like the renamed target in your example about updating the tag. As such, I used a variation on this command because I needed to update the /syn24862084 to match my id, etc. (It may be worth adding to the README in the same section where you talk about "updating your CI/CD workflow" to explicitly call out the docker-compose.yml as another location to update default values.)

docker tag nlpsandbox/date-annotator-example:1.0.0 \ docker.synapse.org/syn24862084/date-annotator-example:1.0.0

I also didn't update the Share Settings since this is just a clone of the demo repository. The instructions indicate you should only update these settings to make the image public, which I'm happy to do for testing but doesn't appear to be essential.

I just made the docker.synapse.org/syn28128744/date-annotator-example-java image publicly available/shared, in case we can use that to help with troubleshooting.

2. Did you submit the Docker image to the queue NLP sandbox - Date Annotator?

Yes. I submitted to the Date Annotator queue. Confirmation for this step is that I see the INVALID status for this submission under my Date Annotation Dashboard entries.

[tschaffter]

@paulheider Did you also submitted the docker image of the Python version of the date annotator example? If yes, can you share the link to your docker image with me? If you are comfortable with using Python, working with this version will be easier for me.

[tschaffter]

What I tried:

• Build the docker image of the Java example of the date annotator => submission failed
• Build the docker image of the Python version of the date annotator => submission successful

I suggest to stick with the Python version for now if you are comfortable with Python. I will troubleshoot the Java version when I'm back from vacation end of April.