the joint accuracy on pretrained model is 0.43,can't reach 0.53

[huangcaiyun]

Hi! Thanks for sharing the pretrained model. I have downloaded your pretrained SOM-DST model , but the result of the evaluation is:

op_code: 4, is_gt_op: False, is_gt_p_state: False, is_gt_gen: False
Epoch 0 joint accuracy : 0.43078175895765475
Epoch 0 slot turn accuracy : 0.9664268910603907
Epoch 0 slot turn F1: 0.8821281530381658
Epoch 0 op accuracy : 0.9676212450234745
Epoch 0 op F1 : {'delete': 0.01583635763774332, 'update': 0.7806280915679281, 'dontcare': 0.13675213675213674, 'carryover': 0.9831123351416803}
Epoch 0 op hit count : {'delete': 24, 'update': 6786, 'dontcare': 32, 'carryover': 207041}
Epoch 0 op all count : {'delete': 2995, 'update': 9351, 'dontcare': 280, 'carryover': 208414}
Final Joint Accuracy : 0.2862862862862863
Final slot turn F1 : 0.8881324359127896
Latency Per Prediction : 26.158629 ms

As you know,the joint accuracy result reported in paper was 0.53, the model you shared is the best model?

[aLowMagic]

Hello, can we talk in wechat? I can give you my id

[huangcaiyun]

OK~ What‘s your wechat id?

[aLowMagic]

replyed in e-mail, or you can send yours to my email. you can see it in my self-page

[DSKSD]

@huangcaiyun
Can you give me more details?
(e.g> pytorch version, cuda, etc ...)

[huangcaiyun]

@DSKSD
The version of pytorch is 1.7.0, and the cuda is 11.0. The dependencies is the same as requirement.txt.
After many times in running program, I have got the best model as reported in paper ,the joint accuracy have been reached 0.53 on multiWoz 2.0. I found that I have got the different result from the same test set, the joint accuracy result on the pretrained model you shared is 0.43, and the joint accuracy result on the model I trained is 0.53.
Did you check the model before you upload? Is it possible that the pretrained model you shared is wrong ?

[DSKSD]

@huangcaiyun
Thank you for sharing.
Yes, I double-checked the result before uploading.

It seems there are some differences according to version up.
I'll check the result based on the environment you mentioned.

[asifdadra]

@huangcaiyun
Can you give me more details?
(e.g> pytorch version, cuda, etc ...)

python3.6
pytorch-transformers==1.0.0
torch==1.3.0a0+24ae9b5
wget==3.2

You can use Google Colab for practice with an environment having Python 3.6 and GPU and Batch_Size=16,

[asifdadra]

Hello, can we talk in wechat? I can give you my id

Hello Friend, you have done superb effort in this work. May I also have your wechat ID after sending you email on [email protected].

[zsc19]

op_code: 4, is_gt_op: False, is_gt_p_state: False, is_gt_gen: False
Epoch 0 joint accuracy : 0.43078175895765475
Epoch 0 slot turn accuracy : 0.9664268910603907
Epoch 0 slot turn F1: 0.8821281530381658
Epoch 0 op accuracy : 0.9676212450234745
Epoch 0 op F1 : {'delete': 0.01583635763774332, 'update': 0.7806280915679281, 'dontcare': 0.13675213675213674, 'carryover': 0.9831123351416803}
Epoch 0 op hit count : {'delete': 24, 'update': 6786, 'dontcare': 32, 'carryover': 207041}
Epoch 0 op all count : {'delete': 2995, 'update': 9351, 'dontcare': 280, 'carryover': 208414}
Final Joint Accuracy : 0.2862862862862863
Final slot turn F1 : 0.8881324359127896
Latency Per Prediction : 941.023903 ms

hotel 0.42192948469585434 0.9687201343148802
train 0.7399591558883595 0.9890401633764536
restaurant 0.48922863099374564 0.9775191104934121
attraction 0.53156146179402 0.982267441860475
taxi 0.46153846153846156 0.9725961538461505