Does model training exist in an additive manner #355

CarbonPool · 2020-08-24T15:01:42Z

For the same model, some additional images are added, an image list is generated, and then image data is generated for training. Does the model remember the image data of the last training?

nagadomi · 2020-08-25T05:06:09Z

I don't fully understand the meaning of your question.
You can load the trained model parameters specified by -resume <model_file.t7> option.
Related to: #209

I also use this option for training each noise level.
In noise scale 0-3 training, ${MODEL_DIR}/scale2.0x_model.t7 is loaded as an initial parameter with the -resume option.

waifu2x/appendix/train_cunet_art.sh

Lines 10 to 37 in d5171bc

    
           # scale 
        
           th train.lua -method scale -save_history 1 -model upcunet -downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/scale_test.png -thread ${THREADS} -backend cudnn -oracle_rate 0.0 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 2 -epoch 200 -patches 32 -batch_size 8 > ${MODEL_DIR}/train.log 2>&1  
        
           # noise scale 0 
        
           th train.lua -save_history 1 -model upcunet -method noise_scale -noise_level 0 -style art \ 
        
           -downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/noise_test.jpg -thread ${THREADS} -backend cudnn -oracle_rate 0.1 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 4 -epoch 30 -patches 32 -batch_size 8 \ 
        
           -resume ${MODEL_DIR}/scale2.0x_model.t7 \ 
        
            > ${MODEL_DIR}/train_noise_scale0.log 2>&1  
        
           # noise scale 1 
        
           th train.lua -save_history 1 -model upcunet -method noise_scale -noise_level 1 -style art \ 
        
           -downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/noise_test.jpg -thread ${THREADS} -backend cudnn -oracle_rate 0.1 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 4 -epoch 30 -patches 32 -batch_size 8 \ 
        
           -resume ${MODEL_DIR}/scale2.0x_model.t7 \ 
        
            > ${MODEL_DIR}/train_noise_scale1.log 2>&1  
        
           # noise scale 2 
        
           th train.lua -save_history 1 -model upcunet -method noise_scale -noise_level 2 -style art \ 
        
           -downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/noise_test.jpg -thread ${THREADS} -backend cudnn -oracle_rate 0.1 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 4 -epoch 30 -patches 32 -batch_size 8 \ 
        
           -resume ${MODEL_DIR}/scale2.0x_model.t7 \ 
        
            > ${MODEL_DIR}/train_noise_scale2.log 2>&1  
        
           # noise scale 3 
        
           th train.lua -save_history 1 -model upcunet -method noise_scale -noise_level 3 -style art \ 
        
           -downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/noise_test.jpg -thread ${THREADS} -backend cudnn -oracle_rate 0.1 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 4 -epoch 30 -patches 32 -batch_size 8 -nr_rate 1 \ 
        
           -resume ${MODEL_DIR}/scale2.0x_model.t7 \ 
        
            > ${MODEL_DIR}/train_noise_scale3.log 2>&1

CarbonPool · 2020-08-25T18:29:01Z

I don't fully understand the meaning of your question.
You can load the trained model parameters specified by -resume <model_file.t7> option.
Related to: #209

I also use this option for training each noise level.
In noise scale 0-3 training, ${MODEL_DIR}/scale2.0x_model.t7 is loaded as an initial parameter with the -resume option.

waifu2x/appendix/train_cunet_art.sh

Lines 10 to 37 in d5171bc

# scale

th train.lua -method scale -save_history 1 -model upcunet -downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/scale_test.png -thread ${THREADS} -backend cudnn -oracle_rate 0.0 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 2 -epoch 200 -patches 32 -batch_size 8 > ${MODEL_DIR}/train.log 2>&1

# noise scale 0

th train.lua -save_history 1 -model upcunet -method noise_scale -noise_level 0 -style art \

-downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/noise_test.jpg -thread ${THREADS} -backend cudnn -oracle_rate 0.1 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 4 -epoch 30 -patches 32 -batch_size 8 \

-resume ${MODEL_DIR}/scale2.0x_model.t7 \

> ${MODEL_DIR}/train_noise_scale0.log 2>&1

# noise scale 1

th train.lua -save_history 1 -model upcunet -method noise_scale -noise_level 1 -style art \

-downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/noise_test.jpg -thread ${THREADS} -backend cudnn -oracle_rate 0.1 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 4 -epoch 30 -patches 32 -batch_size 8 \

-resume ${MODEL_DIR}/scale2.0x_model.t7 \

> ${MODEL_DIR}/train_noise_scale1.log 2>&1

# noise scale 2

th train.lua -save_history 1 -model upcunet -method noise_scale -noise_level 2 -style art \

-downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/noise_test.jpg -thread ${THREADS} -backend cudnn -oracle_rate 0.1 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 4 -epoch 30 -patches 32 -batch_size 8 \

-resume ${MODEL_DIR}/scale2.0x_model.t7 \

> ${MODEL_DIR}/train_noise_scale2.log 2>&1

# noise scale 3

th train.lua -save_history 1 -model upcunet -method noise_scale -noise_level 3 -style art \

-downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -model_dir ${MODEL_DIR} -test query/noise_test.jpg -thread ${THREADS} -backend cudnn -oracle_rate 0.1 -max_size 512 -loss aux_lbp -update_criterion loss -crop_size 104 -validation_crops 64 -inner_epoch 4 -epoch 30 -patches 32 -batch_size 8 -nr_rate 1 \

-resume ${MODEL_DIR}/scale2.0x_model.t7 \

> ${MODEL_DIR}/train_noise_scale3.log 2>&1

Can you explain to me the meaning of "save_history"? I haven't seen the detailed parameter document about train.lua. If the "save_history" parameter is not used, does the learning rate start again when I train this model again?

nagadomi · 2020-08-25T21:07:35Z

When -save_history 1 is specified, all model files with updated best validation scores will be saved with different filenames.
When -save_history 0 is specified(default) , only the model file with the highest validation score will be saved.

leraning rate

The learning rate starts from the default value (0.00025) unless you specify the -learning_rate option. If you want fine tuning, specify a small value.

CarbonPool · 2020-08-26T07:01:48Z

This is the output of each round of training:

[======================================= 134528/134528 ===============================>] Tot: 19m2s | Step: 8ms
{
loss : 0.00035312999208403
MSE : 0.00071552308758262
PSNR : 32.32026436039
}

validation

[======================================= 21120/21120 =================================>] Tot: 1m37s | Step: 4ms

model has updated
Batch-wise PSNR: 33.284603943916, loss: 0.00028738800909366, MSE: 0.00048531165195975, best: 0.00048531165195975

update

Does the "best" field represent the latest learning rate?

nagadomi · 2020-08-26T07:26:56Z

No. It is lowest MSE(mean squared error).
The learning rate is displayed in the cosole output for each round.

# 2	
learning rate: 0.00024853029126955

CarbonPool · 2020-08-26T07:31:48Z

No. It is lowest MSE(mean squared error).
The learning rate is displayed in the cosole output for each round.
# 2	
learning rate: 0.00024853029126955	

I didn't see the "learning rate" field, this is the output from the beginning of the script:

th train.lua -model_dir models/my_model -method noise -noise_level 2 -learning_rate 0.00029588596682772 -test images/3688-gigapixel-scale-1_50x.tif
{
grayscale : false
thread : -1
name : "user"
loss : "huber"
random_erasing_rect_min : 8
use_transparent_png : false
model : "vgg_7"
random_erasing_rate : 0
random_pairwise_negate_x_rate : 0
resume_epoch : 1
downsampling_filters :
{
1 : "Box"
2 : "Lanczos"
3 : "Sinc"
}
resume : ""
crop_size : 48
random_pairwise_rotate_min : -6
random_blur_size : "3,5"
random_pairwise_scale_max : 1.176
random_blur_rate : 0
random_pairwise_negate_rate : 0
nr_rate : 0.65
oracle_drop_rate : 0.5
inner_epoch : 4
invert_x : false
epoch : 50
update_criterion : "mse"
pairwise_flip : true
jpeg_chroma_subsampling_rate : 0.5
image_list : "./data/image_list.txt"
oracle_rate : 0.1
active_cropping_tries : 10
backend : "cunn"
random_pairwise_scale_min : 0.85
active_cropping_rate : 0.5
batch_size : 16
random_unsharp_mask_rate : 0
max_size : 256
validation_crops : 200
plot : false
random_erasing_rect_max : 32
resize_blur_max : 1.05
gpu :
{
1 : 1
}
random_pairwise_scale_rate : 0
random_pairwise_rotate_rate : 0
random_color_noise_rate : 0
validation_filename_split : false
images : "./data/images.t7"
model_file : "models/my_model/noise2_model.t7"
resize_blur_min : 0.95
padding_y_zero : false
test : "images/3688-gigapixel-scale-1_50x.tif"
learning_rate_decay : 3e-07
method : "noise"
save_history : false
color : "rgb"
seed : 11
pairwise_y_binary : false
model_dir : "models/my_model"
style : "art"
data_dir : "./data"
noise_level : 2
random_half_rate : 0
validation_rate : 0.05
random_overlay_rate : 0
max_training_image_size : -1
padding : 0
padding_x_zero : false
scale : 2
random_pairwise_rotate_max : 6
learning_rate : 0.00029588596682772
random_blur_sigma_max : 1
patches : 64
random_erasing_n : 1
random_blur_sigma_min : 0.5
}

0 small images are removed

make validation-set

load .. 2102============================= 100/110 =============================>........] ETA: 1s81ms | Step: 108ms

1

resampling

[======================================= 2102/2102 ===================================>] Tot: 48s274ms | Step: 23ms

update

[======================================= 75969/134528 .................................] ETA: 8m29s | Step: 8ms
[======================================= 134528/134528 ===============================>] Tot: 19m2s | Step: 8ms
{
loss : 0.00035312999208403
MSE : 0.00071552308758262
PSNR : 32.32026436039
}

validation

[======================================= 21120/21120 =================================>] Tot: 1m37s | Step: 4ms

model has updated
Batch-wise PSNR: 33.284603943916, loss: 0.00028738800909366, MSE: 0.00048531165195975, best: 0.00048531165195975

update

[======================================= 134528/134528 ===============================>] Tot: 20m17s | Step: 9ms
{
loss : 0.00024134805397074
MSE : 0.00046385142216102
PSNR : 33.535783961943
}

validation

[======================================= 21120/21120 =================================>] Tot: 1m24s | Step: 3ms
Batch-wise PSNR: 33.074105057769, loss: 0.00022798939562222, MSE: 0.00050555085663752, best: 0.00048531165195975

update

[================>...................... 27969/134528 .................................] ETA: 14m3s | Step: 7ms
[====================>.................. 34369/134528 .................................] ETA: 13m15s | Step: 7ms
[=================================>..... 54369/134528 .................................] ETA: 11m50s | Step: 8ms

CarbonPool · 2020-08-26T07:44:38Z

Sorry, I think I misunderstood the meaning, I thought that each "validation" represents a round, it seems that the next round is really long

nagadomi · 2020-08-26T09:08:57Z

It is shown after # 2.
-inner_epoch 4 by default, so it takes 1 round with 4 update+validation.

CarbonPool · 2020-08-26T09:21:57Z

It is shown after # 2.
-inner_epoch 4 by default, so it takes 1 round with 4 update+validation.

Thank you very much for your warm help, I have learned a lot.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Does model training exist in an additive manner #355

Does model training exist in an additive manner #355

CarbonPool commented Aug 24, 2020

nagadomi commented Aug 25, 2020

CarbonPool commented Aug 25, 2020

nagadomi commented Aug 25, 2020

CarbonPool commented Aug 26, 2020

nagadomi commented Aug 26, 2020

CarbonPool commented Aug 26, 2020

CarbonPool commented Aug 26, 2020

nagadomi commented Aug 26, 2020

CarbonPool commented Aug 26, 2020

Does model training exist in an additive manner #355

Does model training exist in an additive manner #355

Comments

CarbonPool commented Aug 24, 2020

nagadomi commented Aug 25, 2020

CarbonPool commented Aug 25, 2020

nagadomi commented Aug 25, 2020

CarbonPool commented Aug 26, 2020

validation

update

nagadomi commented Aug 26, 2020

CarbonPool commented Aug 26, 2020

make validation-set

1

resampling

update

validation

update

validation

update

CarbonPool commented Aug 26, 2020

nagadomi commented Aug 26, 2020

CarbonPool commented Aug 26, 2020