Mesh style artifacts on new scale 2x model

[chungexcy]

Today when I try new scale model, I found some images showing some 45 degree mesh on the surface where should be smooth. I upload two sets of sample to here. The mesh is on the her face and white sleeve.

Is this what you mentioned

"Now this model has problem in rare case(small pixel art). I will release the fixed versin in a few months."

in another issue? Thanks.

The new model preview server seems down. I didn't test them on your server.

[nagadomi]

Yes. That is known issue what i mentioned and tried to fix.
In that image, I think that cause is video encoding noise. (scale2x is trained on only noisefree images)
noise1_scale2x does not make artifacts like that.
And, I am planning to add noise0_scale2x model that is lower noise reduction filter than noise1_scale2x.

[chungexcy]

Thanks for your explanation. Noise-free training output with downscaled input, it dose make sense the network never see such slight video coding noises.
I doom in the 2x image and the mesh shows a pixel pattern of 4x4. Deconv seems to be more aggressive on this area than your old model.
For noise0_scale2x you mentioned, are you considering to train it with slight noisy output images or just with less jpeg compressed input images on clean output, in order to fix it as well as to low the noise reduction than noise1?

[nagadomi]

just with less jpeg compressed input images on clean output

this. current noise1 supports jpeg image input that jpeg compression quality is uniform(65, 85). and a user mentioned me 'we need more weak noise reduction filter'. I agree that because of jpeg compression quality, uniform(65, 85) is not very high quality. I will add noise0 that supports jpeg quality around uniform(85,95)? (i am still not considering it. I never add noise**-1** filter).

[YukiSakuma]

This happens on waifu2x caffe too (1.1.6) but this time it happens even greater than 2x scaling especially in UpRGB. With RGB noise then RGB scaling there's no mesh artifact
Example
UpRGB denoise with TTA then UpRGB scale = mesh artifact
RGB denoise with TTA then UpRGB scale = mesh artifactt
RGB denoise with TTA then RGB scale = no mesh artifact
UpRGB denoise with TTA then RGB scale = no mesh artifact
I am not an expert but based on my own experiment above maybe the problem is the UpRGB scaling that causes mesh artifacts?
My test image came from Pixiv though. Mesh artifact does not seem to happen on the Miku test image
Maybe I should create an issue?

[nagadomi]

UpRGB denoise with TTA then UpRGB scale = mesh artifact

Could you link image? or send email to me.

[YukiSakuma]

http://www.pixiv.net/member_illust.php?mode=manga&illust_id=55542319
I used the first image and used x5 scaling. The mesh artifact is located above her right eye but it goes all the way up to her right forehead under the hair shadow.
There's also a mesh artifact on her hair clip (color pink)

[nagadomi]

1. 2x with noise0_scale2x: no mesh artifact
2. 4x with scale2x: no mesh artifact (or mesh artifact a very little)
3. 8x with scale2x: mesh artifact

It seems that mesh artifact is amplified by repeatedly scaling with scale2x only model.
current scale2x model does not support repeatedly usage, actually. (It is not optimized for that)

[YukiSakuma]

2x with noise0_scale2x: no mesh artifact
4x with scale2x: no mesh artifact (or mesh artifact a very little)
8x with scale2x: mesh artifact
It seems that mesh artifact is amplified by repeatedly scaling with scale2x only model.
current scale2x model does not support repeatedly usage, actually. (It is not optimized for that)

Does that only apply to UpRGB scaling? Because I tried UpRGB denoise then RGB scale x8 I see no mesh artifact (at least to me). I might upload my results I have slow internet right now.

[nagadomi]

I used only UpRGB model.

[nagadomi]

Maybe here is more better for 8x.

1. noise0_scale2x (2x)
2. noise0_scale2x (4x)
3. noise0_scale2x (8x)

or

1. noise0_scale2x (2x)
2. scale2x (4x)
3. noise0_scale2x (8x)

no mesh artifact and result is very similar to above 8x scaling.

[YukiSakuma]

The mesh disappeared on the 1st method while the 2nd method it is not noticeable unless you really really look closely.
But if possible I really want to use denoising only once but for now it seems unlikely 😭 .

[nagadomi]

4x and 8x are not fully supported because this is waifu2x 😭

[nagadomi]

I updated 2x model 9563d84. (scale2.0x_model.json is for waifu2x-caffe)
It seems that this issue is fixed.

EDIT: I will update all 2x models, Please wait for few weeks ...

[chungexcy]

Saw your new updated models. Do these new models without bias showing better performance in PSNR?

[nagadomi]

No, I thought that the bias in last layer is strong correlation between the mean color of dataset and superresolution task should be more important the input image. so I removed it. but it does not improves performance (PSNR of bias/nobias are very close).
Other way, I decreased -oracle_rate (it is parameter to proactively reduce loss of details). It seems that this change reduced the mesh artifact.
Sorry, I am not able to describe correctly this in my poor english 😞 . I am manually controlling the dataset distribution using -active_cropping_rate and -oracle_rate. active_cropping rejects the single color region of images (fanarts has lots of single color background region). -oracle_rate is similar to -active_cropping_rate but it is using training loss of patch in previous epoch. I you are wondering this, please read the code.

[chungexcy]

Thanks so much for your explanation. I see your point 🔑 . -oracle_rate seems to play a role of fine-tuning the Network on those hard case samples.
At first I thought you solved this artifacts by removing the bias on deconv layer and asked about the performance. Now I understand you used a better learning method to solve it.
So I guess, after killing this kind of artifact, does this new model with better -oracle_rate achieve a higher PSNR performance?

[nagadomi]

PSNR of latest scale2.0x_model is higher than that of before one. I tried same change to train noise*_scale models but it did not improve PSNR and visual quality of the result image.

-oracle_rate, -active_cropping_rate

Those might be domain specific heuristic parameter. fanart/manga images are sparse(lots of white background area and few lines ). When using uniform patch sampling, average of pixel-wise loss of minibatches are the zero or low, and unstable.

[nagadomi]

This issue is still not fully solved 😢

[nagadomi]

Now I've updated upconv_7/art/scale2.0x model. Maybe this issue is fixed ..

[YukiSakuma]

Still need to wait for w2x-caffe to be updated though... 👍

[nagadomi]

Maybe you can manually update waifu2x-caffe model with:

1. open bin/models/upconv_7_anime_style_art_rgb folder
2. download scale2.0x_model.json from https://github.com/nagadomi/waifu2x/blob/master/models/upconv_7/art/scale2.0x_model.json and overwrite.
3. delete scale2.0x_model.json.caffemodel
4. restart waifu2x-caffe

[YukiSakuma]

I updated and there's no more mesh artifact on the pixiv image 😌 Thanks.
but I noticed something
I tested 8x scaling and using denoising only once with TTA:

1. 2x scaling 3 times
2. 8x scaling in one go
   Why is it 2x scaling 3 times output file size is smaller than 8x scaling in one go?