SpanProto evaluation from checkpoint doesn't reproduce the f1 score

[sayef]

Hi Team SpanProto,

As described in the README file of span-proto directory, I trained with few-nerd 5way 5shot inter dataset. The prediction result shows ~0.82 f1 score as presented in the paper. However, when I am trying to use the trained checkpoint for prediction only i.e. adding --resume_from_checkpoint=./outputs/bert-base-uncased or --resume_from_checkpoint=./outputs/bert-base-uncased/checkpoint-7000 etc and remove --do_train argument, the result is bad like ~0.1 f1 score. Could you please tell me how can I reproduce the result of the test set using the trained checkpoint once the training is complete?

Thanks,
sayef

[chywang]

@wjn1996 would u check this pls

[sayef]

any update on this?

[swaggy66]

Hi Team SpanProto,

As described in the README file of span-proto directory, I trained with few-nerd 5way 5shot inter dataset. The prediction result shows ~0.82 f1 score as presented in the paper. However, when I am trying to use the trained checkpoint for prediction only i.e. adding --resume_from_checkpoint=./outputs/bert-base-uncased or --resume_from_checkpoint=./outputs/bert-base-uncased/checkpoint-7000 etc and remove --do_train argument, the result is bad like ~0.1 f1 score. Could you please tell me how can I reproduce the result of the test set using the trained checkpoint once the training is complete?

Thanks, sayef

What is your recall？always 1？