Training time is faster in beginning

[JLDDER]

I train my custom dataset with YOLOP with tesla V100, I found that it training speed is very fast in first but it will get slow after 40 times. May I ask if this is normal or I ignore some settings? Thanks for any suggestions.

Epoch: [5][0/808] Time 7.420s (7.420s) Speed 6.5 samples/s Data 5.233s (5.233s) Loss 0.40308 (0.40308)
Epoch: [5][20/808] Time 113.373s (60.213s) Speed 0.4 samples/s Data 111.287s (58.143s) Loss 0.43158 (0.41728)
Epoch: [5][40/808] Time 226.024s (115.841s) Speed 0.2 samples/s Data 223.994s (113.749s) Loss 0.43076 (0.41315)
Epoch: [5][60/808] Time 334.304s (170.884s) Speed 0.1 samples/s Data 332.240s (168.793s) Loss 0.39735 (0.41089)
Epoch: [5][80/808] Time 448.564s (226.228s) Speed 0.1 samples/s Data 446.482s (224.125s) Loss 0.40302 (0.40986)
Epoch: [5][100/808] Time 561.691s (281.968s) Speed 0.1 samples/s Data 559.411s (279.865s) Loss 0.39656 (0.40988)

[andlyu]

I had the same question. I noticed that this occurred both with a pre-trained model and training from scratch, so I assume it has something to do with the way training is set up/memory. Though wasn't sure exactly why it is happening or if its something that can be optimized.

[junhocho]

Seems the code calculate time taken of data_time and batch_time with time.time() - start.

YOLOP/lib/core/function.py

Lines 48 to 95 in d37e600

	# switch to train mode
	model.train()
	start = time.time()
	for i, (input, target, paths, shapes) in enumerate(train_loader):
	intermediate = time.time()
	#print('tims:{}'.format(intermediate-start))
	num_iter = i + num_batch * (epoch - 1)
	if num_iter < num_warmup:
	# warm up
	lf = lambda x: ((1 + math.cos(x * math.pi / cfg.TRAIN.END_EPOCH)) / 2) * \
	(1 - cfg.TRAIN.LRF) + cfg.TRAIN.LRF # cosine
	xi = [0, num_warmup]
	# model.gr = np.interp(ni, xi, [0.0, 1.0]) # iou loss ratio (obj_loss = 1.0 or iou)
	for j, x in enumerate(optimizer.param_groups):
	# bias lr falls from 0.1 to lr0, all other lrs rise from 0.0 to lr0
	x['lr'] = np.interp(num_iter, xi, [cfg.TRAIN.WARMUP_BIASE_LR if j == 2 else 0.0, x['initial_lr'] * lf(epoch)])
	if 'momentum' in x:
	x['momentum'] = np.interp(num_iter, xi, [cfg.TRAIN.WARMUP_MOMENTUM, cfg.TRAIN.MOMENTUM])
	data_time.update(time.time() - start)
	if not cfg.DEBUG:
	input = input.to(device, non_blocking=True)
	assign_target = []
	for tgt in target:
	assign_target.append(tgt.to(device))
	target = assign_target
	with amp.autocast(enabled=device.type != 'cpu'):
	outputs = model(input)
	total_loss, head_losses = criterion(outputs, target, shapes,model)
	# print(head_losses)
	# compute gradient and do update step
	optimizer.zero_grad()
	scaler.scale(total_loss).backward()
	scaler.step(optimizer)
	scaler.update()
	if rank in [-1, 0]:
	# measure accuracy and record loss
	losses.update(total_loss.item(), input.size(0))
	# _, avg_acc, cnt, pred = accuracy(output.detach().cpu().numpy(),
	# target.detach().cpu().numpy())
	# acc.update(avg_acc, cnt)
	# measure elapsed time
	batch_time.update(time.time() - start)

reseting start with time.time() at the end of for i, (input, target, paths, shapes) in enumerate(train_loader): seems right.

here is my example output when corrected.

Epoch: [1][265/1459]    Time 0.979s (1.027s)    Speed 49.0 samples/s    Data 0.000s (0.015s)    Loss 0.63933 (0.65864)
Epoch: [1][266/1459]    Time 0.981s (1.026s)    Speed 48.9 samples/s    Data 0.000s (0.015s)    Loss 0.65143 (0.65861)
Epoch: [1][267/1459]    Time 0.968s (1.026s)    Speed 49.6 samples/s    Data 0.000s (0.015s)    Loss 0.64569 (0.65857)
Epoch: [1][268/1459]    Time 0.980s (1.026s)    Speed 49.0 samples/s    Data 0.000s (0.015s)    Loss 0.65163 (0.65854)
Epoch: [1][269/1459]    Time 0.956s (1.026s)    Speed 50.2 samples/s    Data 0.000s (0.015s)    Loss 0.65266 (0.65852)
Epoch: [1][270/1459]    Time 0.953s (1.025s)    Speed 50.3 samples/s    Data 0.000s (0.015s)    Loss 0.64746 (0.65848)
Epoch: [1][271/1459]    Time 0.967s (1.025s)    Speed 49.6 samples/s    Data 0.000s (0.015s)    Loss 0.64903 (0.65844)
Epoch: [1][272/1459]    Time 0.961s (1.025s)    Speed 49.9 samples/s    Data 0.000s (0.015s)    Loss 0.65219 (0.65842)
Epoch: [1][273/1459]    Time 1.008s (1.025s)    Speed 47.6 samples/s    Data 0.000s (0.015s)    Loss 0.64192 (0.65836)
Epoch: [1][274/1459]    Time 0.968s (1.025s)    Speed 49.6 samples/s    Data 0.000s (0.015s)    Loss 0.64838 (0.65832)
Epoch: [1][275/1459]    Time 0.963s (1.025s)    Speed 49.8 samples/s    Data 0.000s (0.015s)    Loss 0.64337 (0.65827)
Epoch: [1][276/1459]    Time 0.976s (1.024s)    Speed 49.2 samples/s    Data 0.000s (0.015s)    Loss 0.64825 (0.65823)
Epoch: [1][277/1459]    Time 0.977s (1.024s)    Speed 49.1 samples/s    Data 0.000s (0.015s)    Loss 0.64206 (0.65817)
Epoch: [1][278/1459]    Time 0.971s (1.024s)    Speed 49.4 samples/s    Data 0.000s (0.015s)    Loss 0.64710 (0.65814)
Epoch: [1][279/1459]    Time 0.982s (1.024s)    Speed 48.9 samples/s    Data 0.000s (0.015s)    Loss 0.64195 (0.65808)
Epoch: [1][280/1459]    Time 0.971s (1.024s)    Speed 49.4 samples/s    Data 0.000s (0.015s)    Loss 0.64206 (0.65802)
Epoch: [1][281/1459]    Time 1.001s (1.024s)    Speed 48.0 samples/s    Data 0.000s (0.015s)    Loss 0.64064 (0.65796)
Epoch: [1][282/1459]    Time 1.008s (1.024s)    Speed 47.6 samples/s    Data 0.000s (0.015s)    Loss 0.64724 (0.65792)
Epoch: [1][283/1459]    Time 0.978s (1.023s)    Speed 49.1 samples/s    Data 0.000s (0.014s)    Loss 0.64849 (0.65789)
Epoch: [1][284/1459]    Time 1.036s (1.023s)    Speed 46.3 samples/s    Data 0.000s (0.014s)    Loss 0.64519 (0.65784)
Epoch: [1][285/1459]    Time 0.966s (1.023s)    Speed 49.7 samples/s    Data 0.000s (0.014s)    Loss 0.64414 (0.65780)
Epoch: [1][286/1459]    Time 0.966s (1.023s)    Speed 49.7 samples/s    Data 0.000s (0.014s)    Loss 0.64325 (0.65774)

[HunterShinobiTitan]

+Hi @junhocho, Can you provide the code to correct the output of the time, thanks for help