Inconsistent performance on REC task

[ZhanYang-nwpu]

The performance of Shikra on the dataset of REC task is quite surprising.
I am trying to get the shikra-7b model by using vicuna-7b as the base model and using the shikra-7b-delta-v1 as the delta model.

I evaluate the shikra-7b model on RefCOCO testA and RefCOCO testB, but only get 79.64% and 64.54% overall accuracy.
It does not match the performance on the Table 3.

Do you have any suggestion?

[niiickZ]

Hi, I'm facing the same problem when reproducing the method. Have you solved it?

[ZhanYang-nwpu]

Hi, I'm facing the same problem when reproducing the method. Have you solved it?

I'm sorry, but I still haven't solved the problem. I was confused. I didn't use the shikra model after that.

[Yeemkt]

The performance of Shikra on the dataset of REC task is quite surprising. I am trying to get the shikra-7b model by using vicuna-7b as the base model and using the shikra-7b-delta-v1 as the delta model.

I evaluate the shikra-7b model on RefCOCO testA and RefCOCO testB, but only get 79.64% and 64.54% overall accuracy. It does not match the performance on the Table 3.

Do you have any suggestion?

can u tell me how to evaluate the modle on refcoco or other custom datasets.
Thanks!

[sunsmarterjie]

Could you provide the test code on refcoco?