Cannot reproduce the results on UHDM

[zyxxmu]

Hi authors,
Thanks for the great work! I'm currently trying to reproduce the results of ESDNet on UHDM. Unfortunately, I only got 21.2 dB
PSNR using the training configuration reported in the paper (150 epochs, Bs: 2; Lr: 2e-4; Scheduler: CosineAnnealingWarmRestarts(optimizer, T_0=50, T_mult=1, eta_min=0.000001, last_epoch=-1) provided in your repo; loss: multi_VGGPerceptualLoss(lam=1, lam_p=1) provided in your repo). I would appreciate any suggestions being given.

[XinYu-Andy]

what’s your patch size and and how do you test it?

[zyxxmu]

Patch size 768, Test code the same as

UHDM/model/model.py

Line 55 in 99771dc

def model_fn(args, data, model, iters):

[XinYu-Andy]

Patch size 768, Test code the same as

UHDM/model/model.py

Line 55 in 99771dc

def model_fn(args, data, model, iters):

Can you provide some output images and their individual metrics ? (e.g. images compared in our paper)

[zyxxmu]

Thanks for your reply. The PSNR difference is fixed by replacing my test code for PSNR (skimage.metrics.peak_signal_noise_ratio) with yours

UHDM/utils/metric.py

Line 23 in 99771dc

self.fast_psnr = PSNR()

Your evaluation code for PSNR gives 22.12 dB (similar to the paper), but using skimage.metrics.peak_signal_noise_ratio only gives 21.02 dB. I think the difference may mainly come from converting output fp32 tensors to int8 image arrays or not.

[zyxxmu]

Hi authors, I also trained ESDNet on FHDMi following the configurations given in this repo. (150 epochs, Bs: 2; Lr: 2e-4; Patch size 512). This time I got 22.8 dB PSNR, which is far behind 24.5 dB reported in the paper. I have checked the test code and the evaluation func in your repo is also skimage.metrics.peak_signal_noise_ratio on FHDMi. Therefore, I reopen this issue to figure out if there is any mistakes in my implementation. Many thanks.

[XinYu-Andy]

Hi authors, I also trained ESDNet on FHDMi following the configurations given in this repo. (150 epochs, Bs: 2; Lr: 2e-4; Patch size 512). This time I got 22.8 dB PSNR, which is far behind 24.5 dB reported in the paper. I have checked the test code and the evaluation func in your repo is also skimage.metrics.peak_signal_noise_ratio on FHDMi. Therefore, I reopen this issue to figure out if there is any mistakes in my implementation. Many thanks.

Hi,

We are rushing for ICCV. I suggest that you first check if the output of the test images is normal and whether it can achieve the same results as in our paper. If there is no problem with this, please try our test code to see if the metrics are consistent. Our test code is not unified for all datasets in order to align with previous methods. However, your PSNR is quite lower. Did you save the images as JPEG and cause quality loss? Another way I suggest is to use Matlab for testing. Although the numerical values may be slightly different, it should not vary too much. This can be used to verify whether your training is successful.

[zyxxmu]

Thanks for the suggestion. I‘ll keep trying to fix this. BTW, best wishes for your work ! : )