How to convert a cunet model in .t7 format to a caffe model

[CarbonPool]

With the help of waifu2x-caffe, I want to use the trained cunet model on other platforms. I saw make_cnet.py, but I didn’t understand the processing method, how to convert a .t7 file to a .caffemodel file

[nagadomi]

waifu2x-caffe requires 3 types of files to run the model.
info.json: metadata for the model (name, input size and etc.)
*.prototxt: network structure definition for caffe
*.json: pre-trained parameters(weights) for each .prototxt
These files are located in the following directories.
https://github.com/lltcggie/waifu2x-caffe/tree/master/bin/models
info.json and *.prototxt for cunet/upcunet are already under waifu2x-caffe/bin/models/cunet.
You need to convert *.t7 format to *.json format.

waifu2x side: convert *.t7 format to *.json format. See https://github.com/nagadomi/waifu2x/blob/master/tools/export_all.sh

th tools/export_model.lua -i models/my_cunet/noise3_model.t7 -o models/my_cunet/noise3_model.json

waifu2x-caffe side:

cd ~/waifu2x-caffe/
# Copy cunet to my_cunet
cp -r ./bin/models/cunet ./bin/models/my_cunet
# If *.protobin(cache) exists, delete it.
rm ./bin/models/my_cunet/*.protobin
# Replace the original json file with the converted json file
cp ~/waifu2x/models/my_cunet/noise3_model.json ./bin/models/my_cunet/

test run.

./bin/waifu2x-caffe -p cudnn -m noise -n 3 --model_dir ./bin/models/my_cunet/ -i input.png -o output.png

If you want to use GUI, you need to replace the original bin/models/cunet folder, or modify waifu2x-caffe and rebuild.

[nagadomi]

.caffemodel is probably .protobin in waifu2x-caffe.
If you run waifu2x-caffe once, .protobin will be created from .prototxt and .json.

[CarbonPool]

waifu2x-caffe requires 3 types of files to run the model.
info.json: metadata for the model (name, input size and etc.)
*.prototxt: network structure definition for caffe
*.json: pre-trained parameters(weights) for each .prototxt
These files are located in the following directories.
https://github.com/lltcggie/waifu2x-caffe/tree/master/bin/models
info.json and *.prototxt for cunet/upcunet are already under waifu2x-caffe/bin/models/cunet.
You need to convert *.t7 format to *.json format.

waifu2x side: convert *.t7 format to *.json format. See https://github.com/nagadomi/waifu2x/blob/master/tools/export_all.sh

th tools/export_model.lua -i models/my_cunet/noise3_model.t7 -o models/my_cunet/noise3_model.json

waifu2x-caffe side:

cd ~/waifu2x-caffe/
# Copy cunet to my_cunet
cp -r ./bin/models/cunet ./bin/models/my_cunet
# If *.protobin(cache) exists, delete it.
rm ./bin/models/my_cunet/*.protobin
# Replace the original json file with the converted json file
cp ~/waifu2x/models/my_cunet/noise3_model.json ./bin/models/my_cunet/

test run.

./bin/waifu2x-caffe -p cudnn -m noise -n 3 --model_dir ./bin/models/my_cunet/ -i input.png -o output.png

If you want to use GUI, you need to replace the original bin/models/cunet folder, or modify waifu2x-caffe and rebuild.

Thanks, it works

[CarbonPool]

I trained the cunet noise level2 model and added an additional nr_rate value of 0.75. I found that after converting to the caffe model, I found that their output images are inconsistent. The images obtained using waifu2x.lua retain more details and are cleaner than waifu2x-caffe .

Training command line:
th train.lua -model cunet -model_dir models/my_cunetmodel_noise_level2 -method noise -noise_level 2 -resume models/cunet/art/noise3_model.t7 -downsampling_filters "Box,Box,Box,Box,Sinc,Sinc,Sinc,Sinc,Catrom" -save_history 1 -test "images/000022.jpg" -backend cudnn -style art -nr_rate 0.75 -thread 6 -loss aux_lbp -crop_size 108 -validation_crops 76 -patches 24 -batch_size 16 -epoch 24 -max_size 512 -update_criterion loss -oracle_rate 0.1

Conversion command line:
th tools/export_model.lua -i models/my_cunetmodel_noise_level2/noise2_model.t7 -o models/my_cunetmodel_noise_level2/noise2_model.json

waifu2x.lua

waifu2x-caffe

[nagadomi]

Did you delete *.protobin (cache) after replacing json file?
The result of cunet affected by the block size of tiled processing (-crop_size option). Try adding --crop_size 108 option to waifu2x-caffe-cui.

[CarbonPool]

Did you delete *.protobin (cache) after replacing json file?
The result of cunet affected by the block size of tiled processing (-crop_size option). Try adding --crop_size 108 option to waifu2x-caffe-cui.

Yes, I tried adding the --crop_size 108 option and it still had different results.

This is my file directory:

[CarbonPool]

I was worried that it was selected by the default parameters of waifu2x.lua, so I modified crop_size and batch_size, and it seems that there is still a difference.

th waifu2x.lua -model_dir "models/my_cunetmodel_noise_level2" -m noise -noise_level 2 -force_cudnn 1 -crop_size 144 -batch_size 4 -i "cb01.jpg" -o "output.png"

waifu2x-caffe-cui -p cudnn -m noise --noise_level 2 --crop_size 144 --batch_size 4 --model_dir models/my_cunetmodel -i cb01.jpg -o output.png

[nagadomi]

As I recall, waifu2x and waifu2x-caffe have different definitions of crop_size.
waifu2x: input block size = crop_size
waifu2x-caffe: input block size = crop_size + model_offset(28 for cunet) * 2
So, --crop_size 88 in waifu2x-caffe should be the same as -crop_size 144 in waifu2x.

Another possibility is that waifu2x retains ICC Profile and PNG Gamma metadata, which is discarded by waifu2x-caffe. ffmpeg sometimes embeds Gamma parameter in PNG images, make sure that Gamma of the input image is not other than 0.454545.

% identify -verbose 103534627-3422fa80-4eca-11eb-9ab3-388a43801e8e.png | grep Gamma 
  Gamma: 0.454545

If it is still different, there may be some implementation differences...

[CarbonPool]

As I recall, waifu2x and waifu2x-caffe have different definitions of crop_size.
waifu2x: input block size = crop_size
waifu2x-caffe: input block size = crop_size + model_offset(28 for cunet) * 2
So, --crop_size 88 in waifu2x-caffe should be the same as -crop_size 144 in waifu2x.

Another possibility is that waifu2x retains ICC Profile and PNG Gamma metadata, which is discarded by waifu2x-caffe. ffmpeg sometimes embeds Gamma parameter in PNG images, make sure that Gamma of the input image is not other than 0.454545.

% identify -verbose 103534627-3422fa80-4eca-11eb-9ab3-388a43801e8e.png | grep Gamma 
  Gamma: 0.454545

If it is still different, there may be some implementation differences...

Thanks a lot! Setting the crop_size value through the defined calculation worked for me.