waifu2x/appendix/benchmark.md

# Benchmarks

## Photo

Note: waifu2x's photo models was trained on the blending dataset of [kou's photo collection](http://photosku.com/photo/category/%E6%92%AE%E5%BD%B1%E8%80%85/kou/) and [ukbench](http://vis.uky.edu/~stewe/ukbench/).

Note: PSNR in this benchmark uses a [MATLAB's rgb2ycbcr](https://jp.mathworks.com/help/images/ref/rgb2ycbcr.html?lang=en) compatible function (dynamic range [16 235], not [0 255]) for converting grayscale image. I think it's not correct PSNR. But many paper used this metric.

command: 
`th tools/benchmark.lua -dir <dataset_dir> -model1_dir <model_dir> -method scale -filter Catrom -color y -range_bug 1 -tta <0|1> -force_cudnn 1`

### Datasets

BSD100: https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ (100 test images in BSDS300)
Urban100: https://github.com/jbhuang0604/SelfExSR

### 2x - PSNR 

| Dataset/Model | Bicubic       | vgg\_7/photo  | upconv\_7/photo  | upconv\_7l/photo | resnet_14l/photo | 
|---------------|---------------|---------------|------------------|------------------|--------------------|
| BSD100        | 29.558        | 31.427        | 31.640           | 31.749           | 31.847             |
| Urban100      | 26.852        | 30.057        | 30.477           | 30.759           | 31.016             |

### 2x with TTA - PSNR 

Note: TTA is an ensemble technique that is supported by waifu2x. TTA method is 8x slower than non TTA method but it improves PSNR (~+0.1 on photo, ~+0.4 on art).

| Dataset/Model | Bicubic       | vgg\_7/photo  | upconv\_7/photo  | upconv\_7l/photo | resnet_14l/photo | 
|---------------|---------------|---------------|------------------|------------------|--------------------|
| BSD100        | 29.558        | 31.474        | 31.705           | 31.812           | 31.915             |
| Urban100      | 26.852        | 30.140        | 30.599           | 30.868           | 31.162             |

### 2x - benchmark elapsed time (sec)

| Dataset/Model | vgg\_7/photo  | upconv\_7/photo  | upconv\_7l/photo | resnet_14l/photo |
|---------------|---------------|------------------|------------------|--------------------|
| BSD100        | 4.057         | 2.509            | 4.947            | 6.86               |
| Urban100      | 16.349        | 7.083            | 14.178           | 27.87              |

### 2x with TTA - benchmark elapsed time (sec)

| Dataset/Model | vgg\_7/photo  | upconv\_7/photo  | upconv\_7l/photo | resnet_14l/photo |
|---------------|---------------|------------------|------------------|--------------------|
| BSD100        | 36.611        | 20.219           | 42.486           | 60.38              |
| Urban100      | 132.416       | 65.125           | 129.916          | 255.20             |

## Art

command: 
`th tools/benchmark.lua -dir <dataset_dir> -model1_dir <model_dir> -method scale -filter Lanczos -color y -range_bug 1 -tta <0|1> -force_cudnn 1`

### Dataset

art_test: This dataset contains 85 various fan-arts. Sorry, This dataset is private. 

### 2x - PSNR 

| Dataset/Model | Bicubic       | vgg\_7/art  | upconv\_7/art  | upconv\_7l/art | 
|---------------|---------------|-------------|----------------|----------------|
| art_test      | 31.022        | 37.495      | 38.330         | 39.140         |

### 2x with TTA - PSNR 

| Dataset/Model | Bicubic       | vgg\_7/art  | upconv\_7/art  | upconv\_7l/art | 
|---------------|---------------|-------------|----------------|----------------|
| art_test      | 31.022        | 37.777      | 38.677         | 39.510         |

### 2x - benchmark elapsed time (sec)

| Dataset/Model | vgg\_7/art  | upconv\_7/art  | upconv\_7l/art | 
|---------------|-------------|----------------|----------------|
| art_test      | 20.681      | 7.683          | 17.667         |

### 2x with TTA - benchmark elapsed time (sec)

| Dataset/Model | vgg\_7/art  | upconv\_7/art  | upconv\_7l/art | 
|---------------|-------------|----------------|----------------|
| art_test      | 174.674     | 77.716         | 163.932        |
Update benchmark 2016-10-08 23:36:26 +13:00			`# Benchmarks`
Add benchmark results 2016-03-17 21:40:41 +13:00
Update benchmark 2016-10-08 23:36:26 +13:00			`## Photo`
Add comments to describe that benchmark result is outdated. 2016-07-26 06:18:22 +12:00
Update benchmark.md 2016-10-20 14:17:19 +13:00			`Note: waifu2x's photo models was trained on the blending dataset of [kou's photo collection](http://photosku.com/photo/category/%E6%92%AE%E5%BD%B1%E8%80%85/kou/) and [ukbench](http://vis.uky.edu/~stewe/ukbench/).`
update Benchmark.md 2016-10-09 00:01:15 +13:00
update Benchmark.md 2016-10-08 23:54:02 +13:00			`Note: PSNR in this benchmark uses a [MATLAB's rgb2ycbcr](https://jp.mathworks.com/help/images/ref/rgb2ycbcr.html?lang=en) compatible function (dynamic range [16 235], not [0 255]) for converting grayscale image. I think it's not correct PSNR. But many paper used this metric.`
Add benchmark results 2016-03-17 21:40:41 +13:00
Update benchmark 2016-10-08 23:36:26 +13:00			`command:`
			`th tools/benchmark.lua -dir <dataset_dir> -model1_dir <model_dir> -method scale -filter Catrom -color y -range_bug 1 -tta <0\|1> -force_cudnn 1`
Update benchmark.md 2016-05-06 05:40:15 +12:00
Update benchmark 2016-10-08 23:36:26 +13:00			`### Datasets`
Add benchmark results 2016-03-17 21:40:41 +13:00
update Benchmark.md 2016-10-09 00:01:15 +13:00			`BSD100: https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ (100 test images in BSDS300)`
Update benchmark 2016-10-08 23:36:26 +13:00			`Urban100: https://github.com/jbhuang0604/SelfExSR`
Add benchmark results 2016-03-17 21:40:41 +13:00
Update benchmark 2016-10-08 23:36:26 +13:00			`### 2x - PSNR`
Add benchmark results 2016-03-17 21:40:41 +13:00
rename srresnet_12l to resnet_14l because it's not the same as SRResNet 2017-01-21 20:53:10 +13:00			`\| Dataset/Model \| Bicubic \| vgg\_7/photo \| upconv\_7/photo \| upconv\_7l/photo \| resnet_14l/photo \|`
Add pretraind srresnet_12l model; Update benchmark 2016-12-08 10:15:23 +13:00			`\|---------------\|---------------\|---------------\|------------------\|------------------\|--------------------\|`
			`\| BSD100 \| 29.558 \| 31.427 \| 31.640 \| 31.749 \| 31.847 \|`
			`\| Urban100 \| 26.852 \| 30.057 \| 30.477 \| 30.759 \| 31.016 \|`
Add benchmark results 2016-03-17 21:40:41 +13:00
Update benchmark 2016-10-08 23:36:26 +13:00			`### 2x with TTA - PSNR`
Add benchmark results 2016-03-17 21:40:41 +13:00
update Benchmark.md 2016-10-08 23:54:02 +13:00			`Note: TTA is an ensemble technique that is supported by waifu2x. TTA method is 8x slower than non TTA method but it improves PSNR (~+0.1 on photo, ~+0.4 on art).`
Add benchmark results 2016-03-17 21:40:41 +13:00
rename srresnet_12l to resnet_14l because it's not the same as SRResNet 2017-01-21 20:53:10 +13:00			`\| Dataset/Model \| Bicubic \| vgg\_7/photo \| upconv\_7/photo \| upconv\_7l/photo \| resnet_14l/photo \|`
Add pretraind srresnet_12l model; Update benchmark 2016-12-08 10:15:23 +13:00			`\|---------------\|---------------\|---------------\|------------------\|------------------\|--------------------\|`
			`\| BSD100 \| 29.558 \| 31.474 \| 31.705 \| 31.812 \| 31.915 \|`
			`\| Urban100 \| 26.852 \| 30.140 \| 30.599 \| 30.868 \| 31.162 \|`
Add benchmark results 2016-03-17 21:40:41 +13:00
Update benchmark 2016-10-08 23:36:26 +13:00			`### 2x - benchmark elapsed time (sec)`
Add benchmark results 2016-03-17 21:40:41 +13:00
rename srresnet_12l to resnet_14l because it's not the same as SRResNet 2017-01-21 20:53:10 +13:00			`\| Dataset/Model \| vgg\_7/photo \| upconv\_7/photo \| upconv\_7l/photo \| resnet_14l/photo \|`
Add pretraind srresnet_12l model; Update benchmark 2016-12-08 10:15:23 +13:00			`\|---------------\|---------------\|------------------\|------------------\|--------------------\|`
			`\| BSD100 \| 4.057 \| 2.509 \| 4.947 \| 6.86 \|`
			`\| Urban100 \| 16.349 \| 7.083 \| 14.178 \| 27.87 \|`
Add benchmark results 2016-03-17 21:40:41 +13:00
Update benchmark 2016-10-08 23:36:26 +13:00			`### 2x with TTA - benchmark elapsed time (sec)`

rename srresnet_12l to resnet_14l because it's not the same as SRResNet 2017-01-21 20:53:10 +13:00			`\| Dataset/Model \| vgg\_7/photo \| upconv\_7/photo \| upconv\_7l/photo \| resnet_14l/photo \|`
Add pretraind srresnet_12l model; Update benchmark 2016-12-08 10:15:23 +13:00			`\|---------------\|---------------\|------------------\|------------------\|--------------------\|`
			`\| BSD100 \| 36.611 \| 20.219 \| 42.486 \| 60.38 \|`
			`\| Urban100 \| 132.416 \| 65.125 \| 129.916 \| 255.20 \|`
Update benchmark 2016-10-08 23:36:26 +13:00
			`## Art`

			`command:`
			`th tools/benchmark.lua -dir <dataset_dir> -model1_dir <model_dir> -method scale -filter Lanczos -color y -range_bug 1 -tta <0\|1> -force_cudnn 1`

			`### Dataset`

			`art_test: This dataset contains 85 various fan-arts. Sorry, This dataset is private.`

			`### 2x - PSNR`

			`\| Dataset/Model \| Bicubic \| vgg\_7/art \| upconv\_7/art \| upconv\_7l/art \|`
			`\|---------------\|---------------\|-------------\|----------------\|----------------\|`
			`\| art_test \| 31.022 \| 37.495 \| 38.330 \| 39.140 \|`

			`### 2x with TTA - PSNR`

			`\| Dataset/Model \| Bicubic \| vgg\_7/art \| upconv\_7/art \| upconv\_7l/art \|`
			`\|---------------\|---------------\|-------------\|----------------\|----------------\|`
			`\| art_test \| 31.022 \| 37.777 \| 38.677 \| 39.510 \|`

			`### 2x - benchmark elapsed time (sec)`

			`\| Dataset/Model \| vgg\_7/art \| upconv\_7/art \| upconv\_7l/art \|`
			`\|---------------\|-------------\|----------------\|----------------\|`
			`\| art_test \| 20.681 \| 7.683 \| 17.667 \|`

			`### 2x with TTA - benchmark elapsed time (sec)`

			`\| Dataset/Model \| vgg\_7/art \| upconv\_7/art \| upconv\_7l/art \|`
			`\|---------------\|-------------\|----------------\|----------------\|`
			`\| art_test \| 174.674 \| 77.716 \| 163.932 \|`
Add benchmark results 2016-03-17 21:40:41 +13:00