The reproduced performances on unc/unc+

[OliverHuang1220]

Hi，
The AP I reproduced on unc and unc+ is only 30%，but it is normal on referit and gref.I tried to download the dataset and code again, but it still doesn’t work.
All experimental environments :CUDA9.2、pytorch1.7，batch_size=32 and train on a single 1080ti.
Is there anything else I should pay attention to？

[hbb1]

Hi, have you tried the scripts I provided of unc and unc+? Or do you modified something else?.
Or have you tested the performance using my provided pretrained model to verified the repo installed correctly?
In my opinion, the performance can't be such weak if you run the code correctly.
Also please pay attention to the testing code and the training curve (you can compare to the curve (lbyl_lstm_unc_batch64.out) I provided.)

[OliverHuang1220]

Hi, I tried the scripts of unc and unc+ use lstm model. The lstm model is normal on unc and unc+,but the bert model is weak.
And the bert model 's loss curve can only drop to around 1, so the bert mdel is very weak.The testing code and the lbylnet is OK.
I am very confused about this.

[zhangyr0114]

Hello, I also have this situation. The lstm model is normal on unc and unc+,but the bert model is weak. What is the reason？

[hbb1]

Hi, @zhangyr0114
I tested this repo and reproduced the results. I didn't encounter the situation.
I guess there could be something wrong with loading the pre-trained BERT weights.
Perhaps you can check the training log I provided for debugging. Also, check if it is caused by dependencies.
Here are some related libraries I used in this repo.

pytorch-pretrained-bert 0.6.2
torch                   1.7.1+cu110
torchvision             0.8.2

[SharonCytheria]

@OliverHuang1220 hello, I also have this problem. And I'm curious about the scripts you guys are talking about here. What exactly the scripts are?

[OliverHuang1220]

scripts:lbyl_bert_referit/unc/unc+/gref_batch64 are have this problem.

[SharonCytheria]

@OliverHuang1220 I see. Did you address this problem? Or is there anything I can do to address it ?

[OliverHuang1220]

I tried to change the machine and reconfigure the environment, but it didn't solve the problem. By the way, I was wrong just now. Only UNC and unc+ data have problems when using Bert.:)

[SharonCytheria]

@OliverHuang1220 One more question. Did you install the landmark feature convolution successfully? I thought it is a little bit tricky to install it and I had some difficulty installing it when I tried to use 3090 (now I'm using 2080 tho)

[OliverHuang1220]

I followed the steps as readme, and there was no problem. i us 1080Ti and TitanXP, both ok.

[SharonCytheria]

@OliverHuang1220 Thanks bro. I will try to figure it out.

[OliverHuang1220]

It may have something to do with your gcc version

[SharonCytheria]

@OliverHuang1220 really. Currently my gcc version is 6.3.0. is it too advanced? should I change it to 4.9.2

[OliverHuang1220]

I forget my version. I will tell you before tonight.

…

------------------ Original ------------------ From: SharonCytheria ***@***.***> Date: Wed,Jul 20,2022 11:41 AM To: svip-lab/LBYLNet ***@***.***> Cc: Huang Qiang ***@***.***>, Mention ***@***.***> Subject: Re: [svip-lab/LBYLNet] The reproduced performances on unc/unc+ (Issue #9) @OliverHuang1220 really. Currently my gcc version is 6.3.0. is it too advanced? should I change it to 4.9.2 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[SharonCytheria]

@OliverHuang1220 Okay. Thanks a lot

[OliverHuang1220]

my gcc version-5.4.0

[SharonCytheria]

@OliverHuang1220 Thanks

[SharonCytheria]

Truth Suggestion: Do NOT test your model using the parameter at 100 epoch. Find a comparatively good parameter based on the val_loss (generally in the first 10 epoch) and you will get a decent result :)

[OliverHuang1220]

Wow, Let me try！