Benchmarks
Photo
Note: waifu2x's photo models was trained on the blending dataset of kou's photo collection and ukbench.
Note: PSNR in this benchmark uses a MATLAB's rgb2ycbcr compatible function (dynamic range [16 235], not [0 255]) for converting grayscale image. I think it's not correct PSNR. But many paper used this metric.
command:
th tools/benchmark.lua -dir <dataset_dir> -model1_dir <model_dir> -method scale -filter Catrom -color y -range_bug 1 -tta <0|1> -force_cudnn 1
Datasets
BSD100: https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ (100 test images in BSDS300)
Urban100: https://github.com/jbhuang0604/SelfExSR
2x - PSNR
| Dataset/Model |
Bicubic |
vgg_7/photo |
upconv_7/photo |
upconv_7l/photo |
resnet_14l/photo |
| BSD100 |
29.558 |
31.427 |
31.640 |
31.749 |
31.847 |
| Urban100 |
26.852 |
30.057 |
30.477 |
30.759 |
31.016 |
2x - benchmark time (sec)
| Dataset/Model |
vgg_7/photo |
upconv_7/photo |
upconv_7l/photo |
resnet_14l/photo |
| BSD100 |
4.057 |
2.509 |
4.947 |
6.86 |
| Urban100 |
16.349 |
7.083 |
14.178 |
27.87 |
2x with TTA - PSNR
Note: TTA is an ensemble technique that is supported by waifu2x. TTA method is 8x slower than non TTA method but it improves PSNR (~+0.1 on photo, ~+0.4 on art).
| Dataset/Model |
Bicubic |
vgg_7/photo |
upconv_7/photo |
upconv_7l/photo |
resnet_14l/photo |
| BSD100 |
29.558 |
31.474 |
31.705 |
31.812 |
31.915 |
| Urban100 |
26.852 |
30.140 |
30.599 |
30.868 |
31.162 |
2x with TTA - benchmark time (sec)
| Dataset/Model |
vgg_7/photo |
upconv_7/photo |
upconv_7l/photo |
resnet_14l/photo |
| BSD100 |
36.611 |
20.219 |
42.486 |
60.38 |
| Urban100 |
132.416 |
65.125 |
129.916 |
255.20 |
Art
command: See appendix/benchmark.sh
Dataset
art_test: This dataset contains 84 various fan-arts. Sorry, This dataset is private.
2x - PSNR
| Filter/Model |
Bicubic |
vgg_7/art |
upconv_7/art |
cunet/art |
| Lanczos |
31.022 |
37.495 |
38.330 |
39.886 |
| Sinc |
30.947 |
37.722 |
38.538 |
40.312 |
| Catrom(Bicubic) |
30.663 |
37.278 |
37.189 |
40.184 |
| Box |
30.891 |
37.709 |
38.410 |
39.672 |
2x - benchmark time (sec)
| Dataset/Model |
vgg_7/art |
upconv_7/art |
cunet/art |
| art_test |
24.153 |
10.794 |
24.222 |
2x with TTA - PSNR
| Filter/Model |
Bicubic |
vgg_7/art |
upconv_7/art |
cunet/art |
| Lanczos |
31.022 |
37.777 |
38.677 |
40.289 |
| Sinc |
30.947 |
38.005 |
38.883 |
40.707 |
| Catrom(Bicubic) |
30.663 |
37.498 |
37.417 |
40.592 |
| Box |
30.891 |
38.032 |
38.768 |
40.032 |
2x with TTA - benchmark time (sec)
| Dataset/Model |
vgg_7/art |
upconv_7/art |
cunet/art |
| art_test |
207.217 |
99.151 |
211.520 |
