Issue with custom dataset training

[shilpaullas4]

Hi @CoinCheung ,

I am using my own custom dataset to train bisenetv2 model.
Dataset Details :

image size : 1920x1080
Only two values in my label image {0, 127}

Config:

cfg = dict(
    model_type='bisenetv2',
    n_cats=2,
    num_aux_heads=4,
    lr_start=5e-3,
    weight_decay=1e-4,
    warmup_iters=1000,
    max_iter=72500,
    dataset='CustomerDataset',
    im_root='./datasets/custom',
    train_im_anns='./datasets/custom/train.txt',
    val_im_anns='./datasets/custom/val.txt',
    scales=[0.5, 2.],
    cropsize=[512, 512],
    eval_crop=[512, 512],
    eval_scales=[0.5, 0.75, 1.0, 1.25, 1.5, 1.75],
    ims_per_gpu=4,
    eval_ims_per_gpu=1,
    use_fp16=True,
    use_sync_bn=True,
    respth='./res',
)

I have modified base_dataset.py as shown below

def get_image(self, impth, lbpth):
        img = cv2.imread(impth)[:, :, ::-1].copy()
        label = cv2.imread(lbpth, 0)
        _, label = cv2.threshold(label, 0, 255, cv2.THRESH_BINARY)

But I'm getting loss values as zero after few iterations.
Log :

INFO distributed.py(948): Reducer buckets have been rebuilt in this iteration.
INFO logger.py(51): iter: 100/72500, lr: 0.003454, eta: 5:12:27, time: 26.15, loss: 22.3910, loss_pre: 9.5240, loss_aux0: 0.1125, loss_aux1: 0.6861, loss_aux2: 0.9604, loss_aux3: 11.1080
INFO logger.py(51): iter: 200/72500, lr: 0.004348, eta: 5:04:30, time: 24.64, loss: 0.0068, loss_pre: 0.0000, loss_aux0: 0.0068, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000
INFO logger.py(51): iter: 300/72500, lr: 0.005474, eta: 5:03:20, time: 25.08, loss: 0.0035, loss_pre: 0.0000, loss_aux0: 0.0035, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000
INFO logger.py(51): iter: 400/72500, lr: 0.006892, eta: 5:01:46, time: 24.83, loss: 0.0023, loss_pre: 0.0000, loss_aux0: 0.0023, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000
INFO logger.py(51): iter: 500/72500, lr: 0.008676, eta: 5:01:31, time: 25.18, loss: 0.0016, loss_pre: 0.0000, loss_aux0: 0.0016, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000
INFO logger.py(51): iter: 600/72500, lr: 0.010923, eta: 5:00:20, time: 24.75, loss: 0.0011, loss_pre: 0.0000, loss_aux0: 0.0011, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000
INFO logger.py(51): iter: 700/72500, lr: 0.013751, eta: 4:59:14, time: 24.66, loss: 0.0008, loss_pre: 0.0000, loss_aux0: 0.0008, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000
INFO logger.py(51): iter: 800/72500, lr: 0.017311, eta: 4:58:39, time: 24.89, loss: 0.0007, loss_pre: 0.0001, loss_aux0: 0.0006, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000
INFO logger.py(51): iter: 900/72500, lr: 0.021794, eta: 4:58:14, time: 24.99, loss: 0.0005, loss_pre: 0.0000, loss_aux0: 0.0005, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000
INFO logger.py(51): iter: 1000/72500, lr: 0.027437, eta: 4:58:11, time: 25.31, loss: 0.0004, loss_pre: 0.0000, loss_aux0: 0.0003, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000
INFO logger.py(51): iter: 1100/72500, lr: 0.027466, eta: 4:57:49, time: 25.06, loss: 0.0003, loss_pre: 0.0000, loss_aux0: 0.0003, loss_aux1: 0.0000, loss_aux2: 0.0000, loss_aux3: 0.0000

How do I resolve this issue?