waifu2x/appendix/benchmark.md

# Benchmarks

## Photo

Note: waifu2x's photo models was trained on the blending dataset of [kou's photo collection](http://photosku.com/photo/category/%E6%92%AE%E5%BD%B1%E8%80%85/kou/) and [ukbench](http://vis.uky.edu/~stewe/ukbench/).

Note: PSNR in this benchmark uses a [MATLAB's rgb2ycbcr](https://jp.mathworks.com/help/images/ref/rgb2ycbcr.html?lang=en) compatible function (dynamic range [16 235], not [0 255]) for converting grayscale image. I think it's not correct PSNR. But many paper used this metric.

command: 
`th tools/benchmark.lua -dir <dataset_dir> -model1_dir <model_dir> -method scale -filter Catrom -color y -range_bug 1 -tta <0|1> -force_cudnn 1`

### Datasets

BSD100: https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ (100 test images in BSDS300)
Urban100: https://github.com/jbhuang0604/SelfExSR

### 2x - PSNR 

| Dataset/Model | Bicubic       | vgg\_7/photo  | upconv\_7/photo  | upconv\_7l/photo | resnet_14l/photo | 
|---------------|---------------|---------------|------------------|------------------|--------------------|
| BSD100        | 29.558        | 31.427        | 31.640           | 31.749           | 31.847             |
| Urban100      | 26.852        | 30.057        | 30.477           | 30.759           | 31.016             |

### 2x - benchmark time (sec)

| Dataset/Model | vgg\_7/photo  | upconv\_7/photo  | upconv\_7l/photo | resnet_14l/photo |
|---------------|---------------|------------------|------------------|--------------------|
| BSD100        | 4.057         | 2.509            | 4.947            | 6.86               |
| Urban100      | 16.349        | 7.083            | 14.178           | 27.87              |

### 2x with TTA - PSNR 

Note: TTA is an ensemble technique that is supported by waifu2x. TTA method is 8x slower than non TTA method but it improves PSNR (~+0.1 on photo, ~+0.4 on art).

| Dataset/Model | Bicubic       | vgg\_7/photo  | upconv\_7/photo  | upconv\_7l/photo | resnet_14l/photo | 
|---------------|---------------|---------------|------------------|------------------|--------------------|
| BSD100        | 29.558        | 31.474        | 31.705           | 31.812           | 31.915             |
| Urban100      | 26.852        | 30.140        | 30.599           | 30.868           | 31.162             |

### 2x with TTA - benchmark time (sec)

| Dataset/Model | vgg\_7/photo  | upconv\_7/photo  | upconv\_7l/photo | resnet_14l/photo |
|---------------|---------------|------------------|------------------|--------------------|
| BSD100        | 36.611        | 20.219           | 42.486           | 60.38              |
| Urban100      | 132.416       | 65.125           | 129.916          | 255.20             |

## Art

command: See `appendix/benchmark.sh`

### Dataset

art_test: This dataset contains 84 various fan-arts. Sorry, This dataset is private.

### 2x - PSNR 

| Filter/Model   | Bicubic       | vgg\_7/art  | upconv\_7/art  | cunet/art      | 
|----------------|---------------|-------------|----------------|----------------|
| Lanczos        | 31.022        | 37.495      | 38.330         | 39.886         |
| Sinc           | 30.947        | 37.722      | 38.538         | 40.312         |
| Catrom(Bicubic)| 30.663        | 37.278      | 37.189         | 40.184         |
| Box            | 30.891        | 37.709      | 38.410         | 39.672         |

### 2x - benchmark time (sec)

| Dataset/Model | vgg\_7/art  | upconv\_7/art  | cunet/art | 
|---------------|-------------|----------------|----------------|
| art_test      | 24.153      | 10.794         | 24.222         |

### 2x with TTA - PSNR 

| Filter/Model   | Bicubic       | vgg\_7/art  | upconv\_7/art  | cunet/art      | 
|----------------|---------------|-------------|----------------|----------------|
| Lanczos        | 31.022        | 37.777      | 38.677         | 40.289         |
| Sinc           | 30.947        | 38.005      | 38.883         | 40.707         |
| Catrom(Bicubic)| 30.663        | 37.498      | 37.417         | 40.592         |
| Box            | 30.891        | 38.032      | 38.768         | 40.032         |

### 2x with TTA - benchmark time (sec)

| Dataset/Model | vgg\_7/art  | upconv\_7/art  | cunet/art       | 
|---------------|-------------|----------------|----------------|
| art_test      | 207.217     | 99.151         | 211.520        |
Update benchmark 2016-10-08 23:36:26 +13:00			`# Benchmarks`
Add benchmark results 2016-03-17 21:40:41 +13:00
Update benchmark 2016-10-08 23:36:26 +13:00			`## Photo`
Add comments to describe that benchmark result is outdated. 2016-07-26 06:18:22 +12:00
Update benchmark.md 2016-10-20 14:17:19 +13:00			`Note: waifu2x's photo models was trained on the blending dataset of [kou's photo collection](http://photosku.com/photo/category/%E6%92%AE%E5%BD%B1%E8%80%85/kou/) and [ukbench](http://vis.uky.edu/~stewe/ukbench/).`
update Benchmark.md 2016-10-09 00:01:15 +13:00
update Benchmark.md 2016-10-08 23:54:02 +13:00			`Note: PSNR in this benchmark uses a [MATLAB's rgb2ycbcr](https://jp.mathworks.com/help/images/ref/rgb2ycbcr.html?lang=en) compatible function (dynamic range [16 235], not [0 255]) for converting grayscale image. I think it's not correct PSNR. But many paper used this metric.`
Add benchmark results 2016-03-17 21:40:41 +13:00
Update benchmark 2016-10-08 23:36:26 +13:00			`command:`
			`th tools/benchmark.lua -dir <dataset_dir> -model1_dir <model_dir> -method scale -filter Catrom -color y -range_bug 1 -tta <0\|1> -force_cudnn 1`
Update benchmark.md 2016-05-06 05:40:15 +12:00
Update benchmark 2016-10-08 23:36:26 +13:00			`### Datasets`
Add benchmark results 2016-03-17 21:40:41 +13:00
update Benchmark.md 2016-10-09 00:01:15 +13:00			`BSD100: https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ (100 test images in BSDS300)`
Update benchmark 2016-10-08 23:36:26 +13:00			`Urban100: https://github.com/jbhuang0604/SelfExSR`
Add benchmark results 2016-03-17 21:40:41 +13:00
Update benchmark 2016-10-08 23:36:26 +13:00			`### 2x - PSNR`
Add benchmark results 2016-03-17 21:40:41 +13:00
rename srresnet_12l to resnet_14l because it's not the same as SRResNet 2017-01-21 20:53:10 +13:00			`\| Dataset/Model \| Bicubic \| vgg\_7/photo \| upconv\_7/photo \| upconv\_7l/photo \| resnet_14l/photo \|`
Add pretraind srresnet_12l model; Update benchmark 2016-12-08 10:15:23 +13:00			`\|---------------\|---------------\|---------------\|------------------\|------------------\|--------------------\|`
			`\| BSD100 \| 29.558 \| 31.427 \| 31.640 \| 31.749 \| 31.847 \|`
			`\| Urban100 \| 26.852 \| 30.057 \| 30.477 \| 30.759 \| 31.016 \|`
Add benchmark results 2016-03-17 21:40:41 +13:00
update benchmark 2018-11-15 00:30:04 +13:00			`### 2x - benchmark time (sec)`
Fix benchmark format 2018-11-15 00:21:44 +13:00
			`\| Dataset/Model \| vgg\_7/photo \| upconv\_7/photo \| upconv\_7l/photo \| resnet_14l/photo \|`
			`\|---------------\|---------------\|------------------\|------------------\|--------------------\|`
			`\| BSD100 \| 4.057 \| 2.509 \| 4.947 \| 6.86 \|`
			`\| Urban100 \| 16.349 \| 7.083 \| 14.178 \| 27.87 \|`

Update benchmark 2016-10-08 23:36:26 +13:00			`### 2x with TTA - PSNR`
Add benchmark results 2016-03-17 21:40:41 +13:00
update Benchmark.md 2016-10-08 23:54:02 +13:00			`Note: TTA is an ensemble technique that is supported by waifu2x. TTA method is 8x slower than non TTA method but it improves PSNR (~+0.1 on photo, ~+0.4 on art).`
Add benchmark results 2016-03-17 21:40:41 +13:00
rename srresnet_12l to resnet_14l because it's not the same as SRResNet 2017-01-21 20:53:10 +13:00			`\| Dataset/Model \| Bicubic \| vgg\_7/photo \| upconv\_7/photo \| upconv\_7l/photo \| resnet_14l/photo \|`
Add pretraind srresnet_12l model; Update benchmark 2016-12-08 10:15:23 +13:00			`\|---------------\|---------------\|---------------\|------------------\|------------------\|--------------------\|`
			`\| BSD100 \| 29.558 \| 31.474 \| 31.705 \| 31.812 \| 31.915 \|`
			`\| Urban100 \| 26.852 \| 30.140 \| 30.599 \| 30.868 \| 31.162 \|`
Add benchmark results 2016-03-17 21:40:41 +13:00
update benchmark 2018-11-15 00:30:04 +13:00			`### 2x with TTA - benchmark time (sec)`
Update benchmark 2016-10-08 23:36:26 +13:00
rename srresnet_12l to resnet_14l because it's not the same as SRResNet 2017-01-21 20:53:10 +13:00			`\| Dataset/Model \| vgg\_7/photo \| upconv\_7/photo \| upconv\_7l/photo \| resnet_14l/photo \|`
Add pretraind srresnet_12l model; Update benchmark 2016-12-08 10:15:23 +13:00			`\|---------------\|---------------\|------------------\|------------------\|--------------------\|`
			`\| BSD100 \| 36.611 \| 20.219 \| 42.486 \| 60.38 \|`
			`\| Urban100 \| 132.416 \| 65.125 \| 129.916 \| 255.20 \|`
Update benchmark 2016-10-08 23:36:26 +13:00
			`## Art`

Update benchmark 2018-11-14 23:03:10 +13:00			command: See `appendix/benchmark.sh`
Update benchmark 2016-10-08 23:36:26 +13:00
			`### Dataset`

Update benchmark 2018-11-14 23:03:10 +13:00			`art_test: This dataset contains 84 various fan-arts. Sorry, This dataset is private.`
Update benchmark 2016-10-08 23:36:26 +13:00
			`### 2x - PSNR`

Update benchmark 2018-11-14 23:03:10 +13:00			`\| Filter/Model \| Bicubic \| vgg\_7/art \| upconv\_7/art \| cunet/art \|`
			`\|----------------\|---------------\|-------------\|----------------\|----------------\|`
			`\| Lanczos \| 31.022 \| 37.495 \| 38.330 \| 39.886 \|`
			`\| Sinc \| 30.947 \| 37.722 \| 38.538 \| 40.312 \|`
			`\| Catrom(Bicubic)\| 30.663 \| 37.278 \| 37.189 \| 40.184 \|`
			`\| Box \| 30.891 \| 37.709 \| 38.410 \| 39.672 \|`
Update benchmark 2016-10-08 23:36:26 +13:00
update benchmark 2018-11-15 00:30:04 +13:00			`### 2x - benchmark time (sec)`
Fix benchmark format 2018-11-15 00:21:44 +13:00
			`\| Dataset/Model \| vgg\_7/art \| upconv\_7/art \| cunet/art \|`
			`\|---------------\|-------------\|----------------\|----------------\|`
			`\| art_test \| 24.153 \| 10.794 \| 24.222 \|`

Update benchmark 2016-10-08 23:36:26 +13:00			`### 2x with TTA - PSNR`

Update benchmark 2018-11-14 23:03:10 +13:00			`\| Filter/Model \| Bicubic \| vgg\_7/art \| upconv\_7/art \| cunet/art \|`
			`\|----------------\|---------------\|-------------\|----------------\|----------------\|`
			`\| Lanczos \| 31.022 \| 37.777 \| 38.677 \| 40.289 \|`
			`\| Sinc \| 30.947 \| 38.005 \| 38.883 \| 40.707 \|`
			`\| Catrom(Bicubic)\| 30.663 \| 37.498 \| 37.417 \| 40.592 \|`
			`\| Box \| 30.891 \| 38.032 \| 38.768 \| 40.032 \|`

update benchmark 2018-11-15 00:30:04 +13:00			`### 2x with TTA - benchmark time (sec)`
Update benchmark 2016-10-08 23:36:26 +13:00
Fix benchmark format 2018-11-15 00:21:44 +13:00			`\| Dataset/Model \| vgg\_7/art \| upconv\_7/art \| cunet/art \|`
Update benchmark 2016-10-08 23:36:26 +13:00			`\|---------------\|-------------\|----------------\|----------------\|`
Update benchmark 2018-11-14 23:03:10 +13:00			`\| art_test \| 207.217 \| 99.151 \| 211.520 \|`