Benchmark


Overview

For each of the 50 images of our benchmark we provide locations of 20 bounding boxes that are to be denoised individually – thus yielding 1000 bounding boxes in total. For each bounding box we compute PSNR and SSIM values. The final PSNR and SSIM values listed in the tables belows are computed by averaging the per-bounding-box values.

Application of denoising algorithms should be done in one of the following three ways:

  1. Denoising on raw data directly.
  2. Denoising on raw data after a variance stabilizing transformation (VST) was applied. The final denoised image is obtained by inverting the VST.
  3. Denoising on sRGB data directly.

For option 1 and 2 we provide evaluation against RAW and sRGB ground truth, respectively, while for option 3 we evaluate against the sRGB ground truth only.

Anonymous results are shown in italic font.


Results for denoising on raw data without VST

Name Uses VST Denoised on PSNR on raw SSIM on raw PSNR on sRGB SSIM on sRGB RPB Details
N3Net 0 raw 47.5643 0.9767 38.3158 0.9384 1
DSSNet 0 raw 47.3348 0.9784 37.8575 0.9427 0
BM3D 0 raw 46.64 0.9724 37.78 0.9308
EPLL 0 raw 46.31 0.9679 37.16 0.9291
WNNM 0 raw 46.3 0.9707 37.56 0.9313
FoE 0 raw 45.78 0.9666 35.99 0.9042
KSVD 0 raw 45.54 0.9676 36.59 0.9162
TNRD 0 raw 44.97 0.9624 35.57 0.8913
NCSR 0 raw 42.85 0.8527 30.65 0.7025
MLP 0 raw 42.7 0.9395 33.63 0.8829

Results for denoising on raw data with VST

Name Uses VST Denoised on PSNR on raw SSIM on raw PSNR on sRGB SSIM on sRGB RPB Details
BM3D 1 raw 47.15 0.9737 37.86 0.9296
NCSR 1 raw 47.07 0.9688 37.79 0.9233
WNNM 1 raw 47.05 0.9722 37.69 0.926
KSVD 1 raw 46.87 0.9723 37.63 0.9287
EPLL 1 raw 46.86 0.973 37.46 0.9245
MLP 1 raw 45.71 0.9629 36.72 0.9122
TNRD 1 raw 45.7 0.9609 36.09 0.8883
FoE 1 raw 44.12 0.9554 35.92 0.911

Results for denoising on sRGB Data

Name Uses VST Denoised on PSNR on sRGB SSIM on sRGB RPB Details
NLH+ 0 srgb 38.8071 0.952 0
NLH 0 srgb 38.7859 0.9518 0
MCAR 0 srgb 38.4005 0.9452 0
CBDNet 0 srgb 38.0564 0.9421 0.4
DSSNet 0 srgb 38.0391 0.9358 0
TWSC 0 srgb 37.939 0.9403 0
DnCNN+ 0 srgb 37.9018 0.943 0.05
FFDNet+ 0 srgb 37.6107 0.9415 0
MCWNNM 0 srgb 37.379 0.9294 0
LBD 0 srgb 37.3233 0.9431 0
Nonblind_MCD 0 srgb 37.1976 0.9361 0
MCD 0 srgb 37.12 0.9353 0
128-DnCNN tensorflow 0 srgb 37.0343 0.9324 0.04
SCD 0 srgb 36.9453 0.9299 0
KSVD 0 srgb 36.49 0.8978
CIMM 0 srgb 36.0367 0.9136 0
GCBD 0 srgb 35.5796 0.9217
DenoiseNet 0 srgb 35.0802 0.868 0.33
WNNM 0 srgb 34.67 0.8646
FoE 0 srgb 34.62 0.8845
BM3D 0 srgb 34.51 0.8507
UNET based Image denoising 0 srgb 34.3599 0.9221 0.005
MLP 0 srgb 34.23 0.8331
NCSR 0 srgb 34.05 0.8351
TNRD 0 srgb 33.65 0.8306
EPLL 0 srgb 33.51 0.8244
DnCNN 0 srgb 32.4296 0.79