Performance different from reported in paper

[XAVILLA]

Hi, I tried to run the training code with the 1m config but got significantly worse performance compared to the performance reported in the paper. I notice that you mentioned in the paper that batch size 35 was used but in the 1m config you set batch size to be 14. Can you please explain what should I change to train a model that achieves the performance as you reported with 1m config? Thanks!

[johnwlambert]

Hi @XAVILLA, thanks for your interest in our work. I'll double check on this and get back to you.

[XAVILLA]

Hi John. Thanks for the help! Any update so far?

[johnwlambert]

Hi @XAVILLA, which specific model are you training, on which dataset, at which resolution, and using which config?

Could you share the results you are seeing?

[XAVILLA]

Hi @johnwlambert, I used the HRNet+1m+1080 config given in the repo. link. In this config the batch size is set to 14

I run evaluation on the trained model using 360 ss config and obtained mIOU ~58.5, and 1080 ss config and obtained mIOU ~74, both on cityscapes validation set. The mIOU for 1m model was reported in the repo to be ~80 on cityscapes.

Can you take a look at this issue and let me know what should I fix to obtain the performance reported in the paper? Thanks!

[XAVILLA]

Hi @johnwlambert, any update on this issue? My main question is that in the paper you describe that you used a mini-batch size m = 35, but in the training config here the batch size is set to 35. Can you please clarify which batch size was used in your experiments to produce the reported performance? Thanks!