Training and Validation Loss

[gopi-erabati]

I am training the Yolo with MobilenetV1 on VOC07+12 trainval dataset. I loaded Mobilenet with imagenet weights and built Yolo layers on top of it. I freeze the Mobilnet for first 50 epochs and tried to train to get a stabilized loss, after that I unfreeze all layers to train the network further.

I trained up to 105 epochs and still on it, the training loss decreased from 26 to around 12, val loss is around 14, and now further the loss is not decreasing and fluctuating around 12 and 14 respectively, as the loss is on Plateau the learning rate got decreased from 1e-4 to 1e-8.

I want to know:
Is this normal ?
Should I still continue training further ?
If anyone trained on the same dataset, I wanted to know the optimum loss they could reach ? (My each epoch is taking approximately 20mins)

Specifications:
Dataset : VOC 07+12 trainval with 0.1% validation set
Batch size : 16
Input shape : (416, 416)
Training on : Google Colab GPU

[hongsamvo]

Hi @gopi231091
I am training on VOC0712 and got the same result as you. I am wondering this is normal or not too. I hope you can share your experience. Thank you.

[diegolitianfu]

the same result... I want know your final loss and your result on your test set if it possible. Thanks. @hongsamvo @gopi231091

[diegolitianfu]

the same result... I want know your final loss and your result on your test set if it possible. Thanks. @hongsamvo @gopi231091

besides, if you remanbered, is there any other tricks? thank you so much.

[hongsamvo]

@diegolitianfu I did this project for a long time ago, so I can't remember that I solved this issue or not.