Benchmark

Overview

For each of the 50 images of our benchmark we provide locations of 20 bounding boxes that are to be denoised individually – thus yielding 1000 bounding boxes in total. For each bounding box we compute PSNR and SSIM values. The final PSNR and SSIM values listed in the tables belows are computed by averaging the per-bounding-box values.

Application of denoising algorithms should be done in one of the following three ways:

  1. Denoising on raw data directly.
  2. Denoising on raw data after a variance stabilizing transformation (VST) was applied. The final denoised image is obtained by inverting the VST.
  3. Denoising on sRGB data directly.

For option 1 and 2 we provide evaluation against RAW and sRGB ground truth, respectively, while for option 3 we evaluate against the sRGB ground truth only.

Anonymous results are shown in italic font.

Results for denoising on raw data without VST

Name Uses VST Denoised on PSNR on raw SSIM on raw PSNR on sRGB SSIM on sRGB RPB Details
[ECCV2024] DualDn (Intermediate raw out) 0 raw 49.2611 0.9834 40.6999 0.9635 0
Pseudo-ISP 0 raw 49.1308 0.9829 40.3644 0.9606 0
CycleISP 0 raw 49.1251 0.983 40.4987 0.9655 0
TU-Denoiser 0 raw 49.0351 0.9827 40.2087 0.96 0
UPI (Raw) 0 raw 48.8905 0.9824 40.1728 0.9623 0.022
UPI (sRGB) 0 raw 48.8824 0.9821 40.3545 0.9641 0.022
ELMformer 0 raw 48.8663 0.9817 40.0718 0.9556 0
Pseudo-ISP 0 raw 48.8234 0.9816 39.9611 0.9548 0

RU_raw 0 raw 48.7322 0.9812 39.7821 0.9552 0

YOND (with VST) 0 raw 48.5684 0.9814 39.6945 0.9553 -1

TSBDNet 0 raw 48.5522 0.9808 39.5547 0.9502 0.017

PRIDNet_raw 0 raw 48.4839 0.9806 39.4681 0.9522 0

Feature-Align 0 raw 48.2756 0.9808 39.5061 0.9572 0.00079

DCD 0 raw 48.2449 0.9798 39.0964 0.947 -1

N2N 0 raw 48.128 0.9791 39.0101 0.9474 -1

FBI (SIDD) 0 raw 48.0922 0.9789 39.0981 0.9456 0.21

MLRI+1.5CBM3D (VST) 0 raw 47.7573 0.9776 38.3636 0.9379 0

MLRI+1.5CBM3D 0 raw 47.6279 0.977 38.2563 0.9363 0

N3Net 0 raw 47.5643 0.9767 38.3158 0.9384 1

LIChI (with VST) 0 raw 47.3484 0.9754 38.1318 0.9339 1

DSSNet 0 raw 47.3348 0.9784 37.8575 0.9427 0
RCNN+1.5CBM3D (VST) 0 raw 47.2552 0.9765 37.8686 0.933 0

RCNN+1.5CBM3D 0 raw 47.1559 0.976 37.7821 0.9341 0

NL-Ridge (with VST) 0 raw 47.0103 0.9739 37.8212 0.9253 1

aRID 0 raw 46.9878 0.9745 38.1809 0.9385 0
PEWA (with VST) 0 raw 46.721 0.9719 37.6515 0.9262 40

Prompt-SID 0 raw 46.6969 0.9701 37.789 0.9272 -1

BM3D 0 raw 46.64 0.9724 37.78 0.9308

Bitonic MX 0 raw 46.6072 0.9705 38.016 0.9363 1.9

EPLL 0 raw 46.31 0.9679 37.16 0.9291

WNNM 0 raw 46.3 0.9707 37.56 0.9313

blind2unblind 0 raw 45.8676 0.9639 37.198 0.9209 -1

FoE 0 raw 45.78 0.9666 35.99 0.9042

KSVD 0 raw 45.54 0.9676 36.59 0.9162

neighbor 0 raw 45.363 0.9608 36.8735 0.9166 -1

TNRD 0 raw 44.97 0.9624 35.57 0.8913

advect_run05 0 raw 44.4837 0.9526 35.6892 0.8963 0.04

NCSR 0 raw 42.85 0.8527 30.65 0.7025

MLP 0 raw 42.7 0.9395 33.63 0.8829

Results for denoising on raw data with VST

Name Uses VST Denoised on PSNR on raw SSIM on raw PSNR on sRGB SSIM on sRGB RPB Details
BM3D 1 raw 47.15 0.9737 37.86 0.9296

NCSR 1 raw 47.07 0.9688 37.79 0.9233

WNNM 1 raw 47.05 0.9722 37.69 0.926

KSVD 1 raw 46.87 0.9723 37.63 0.9287

EPLL 1 raw 46.86 0.973 37.46 0.9245

MLP 1 raw 45.71 0.9629 36.72 0.9122

TNRD 1 raw 45.7 0.9609 36.09 0.8883

FoE 1 raw 44.12 0.9554 35.92 0.911

Results for denoising on sRGB Data

Name Uses VST Denoised on PSNR on sRGB SSIM on sRGB RPB Details
[ECCV2024] DualDn (Denoise in raw and sRGB domain) 0 srgb 40.5937 0.9661 0

BDE 0 srgb 40.5163 0.9638 0
Pseudo-ISP 0 srgb 40.3221 0.9614 0

DnSwin 0 srgb 40.2898 0.9622 8.5

PNGAN 0 srgb 40.2528 0.9617 0

GMSNet-B 0 srgb 40.237 0.9616 0

GACNet 0 srgb 40.223 0.9597 1
FADNet* 0 srgb 40.2016 0.9595 1

Pseudo-ISP(PT-MWRN) 0 srgb 40.1944 0.9594 0

MDnN_one 0 srgb 40.1903 0.9614 0.4

DMANet 0 srgb 40.1876 0.9592 1

M2AE2Net 0 srgb 40.1808 0.9617 1.1

FADNet 0 srgb 40.1742 0.9592 0

Scaoed_2 0 srgb 40.1663 0.9597 1

MPA-Net 0 srgb 40.1621 0.9593 1

S_TUNet 0 srgb 40.1534 0.9604 8.5

GMSNet-A 0 srgb 40.1503 0.9615 0

Scaoed_1 0 srgb 40.1184 0.9603 1

DCA-DNet 0 srgb 40.102 0.9582 0.5
LGDNet 0 srgb 40.0988 0.9561 0

Res_UNet 0 srgb 40.0612 0.9604 50

PseudoISP(PT-MWRN) 0 srgb 40.054 0.9565 0

PT-MWRN 0 srgb 40.0531 0.9565 0

Restormer 0 srgb 40.0324 0.9564 0

MFU 0 srgb 40.0023 0.9555 0

DeCformer+ 0 srgb 39.9985 0.9553 1
MDDA_former 0 srgb 39.9828 0.9557 0

Uformer 0 srgb 39.9791 0.9554 0

Basemodel 0 srgb 39.9649 0.9587 0

CTHNet2 0 srgb 39.9612 0.955 1
HPDNet 0 srgb 39.9512 0.9565 0.18

TECDNet_S 0 srgb 39.9137 0.955 0.015

MDPA-Net-Enhanced 0 srgb 39.9105 0.9549 1
PCformer 0 srgb 39.9103 0.9552 0.1

Bigk_Rep_attn_128_bs32 0 srgb 39.9036 0.9547 1

LIGN 0 srgb 39.9014 0.9589 0

NBNet 0 srgb 39.8932 0.9554 0

TECDNet_L 0 srgb 39.8861 0.9549 0.015
MIRNet 0 srgb 39.88 0.9563 0

PPDNet 0 srgb 39.8703 0.9551 0
ADFNet 0 srgb 39.8694 0.9552 -1
MDPA-Net 0 srgb 39.8682 0.9549 0
PT-MWRN 0 srgb 39.8447 0.958 0

Bigk_Rep_32_128_bs32 0 srgb 39.8393 0.9536 1

DPDN+ 0 srgb 39.8293 0.9537 0

VIRNet 0 srgb 39.8264 0.9537 0.98

MPRNet 0 srgb 39.8162 0.954 0

MMPDBet_2 0 srgb 39.7817 0.9534 0
ANLD_Baseline 0 srgb 39.7715 0.9538 0.4

AINDNet(S)* 0 srgb 39.7695 0.959 0

Path-Restore-Ext 0 srgb 39.7227 0.9591 0.42

hyh 0 srgb 39.7005 0.9522 1
VDIR+ 0 srgb 39.6941 0.9532 0

DPDN 0 srgb 39.6657 0.9542 0

BCDNet-L 0 srgb 39.6571 0.9532 0
COLA-E 0 srgb 39.6392 0.9538 1

VDIR 0 srgb 39.6337 0.9528 0

CBDNet+ 0 srgb 39.629 0.9571 0.4

DIDN 0 srgb 39.6224 0.9546 0

ACIDN 0 srgb 39.5947 0.9525 -1

SADNet 0 srgb 39.5871 0.9523 0.027

DANet+ 0 srgb 39.5848 0.9545 0.12

allnet 0 srgb 39.5725 0.9519 0.02

Thunder 0 srgb 39.5704 0.9526 1
InvDN 0 srgb 39.5699 0.9522 0

BCDNet 0 srgb 39.5691 0.9528 1
CycleISP 0 srgb 39.5649 0.9564 0

AINDNet(S) 0 srgb 39.526 0.9561 0

AINDNet+TF* 0 srgb 39.5179 0.9522 0

RIDNet_P(mwcnn_DIP_e12) 0 srgb 39.5153 0.9526 0

RU_sRGB 0 srgb 39.5064 0.9528 0

DANet 0 srgb 39.4716 0.9548 0.12

AINDNet+TF (git) 0 srgb 39.4584 0.9522 0

CDNet 0 srgb 39.4482 0.9512 0
GradNet 0 srgb 39.4388 0.954 20

DUBD-R+ 0 srgb 39.435 0.953 0

DRDN 0 srgb 39.4342 0.9531 0

PRIDNet_sRGB 0 srgb 39.4247 0.9528 0.05

ECSCNet 0 srgb 39.4122 0.9506 1.23456

VDN 0 srgb 39.383 0.9518 0.004

DUBD-R 0 srgb 39.3788 0.9526 0

ECSCNet 0 srgb 39.3739 0.9503 1

SADNet (1248) 0 srgb 39.3736 0.9544 0.027

AINDNet+TF 0 srgb 39.3729 0.9505 0

UNet-ND 0 srgb 39.3683 0.9542 0.2

f-CDNet 0 srgb 39.3562 0.9506 0
HI-GAN 0 srgb 39.3226 0.952 0.2

IERD+ 0 srgb 39.3032 0.9531 0.1

SmartDSP 0 srgb 39.302 0.9513 0.01

DHDN 0 srgb 39.2885 0.9515 0

RIDNet 0 srgb 39.2555 0.9528 0.1

MLDN 0 srgb 39.2311 0.9516 0.48

IERD 0 srgb 39.1983 0.9524 0.1

ATDNet 0 srgb 39.1901 0.9526 0.01

CBDNet + XPrior 0 srgb 39.1518 0.9556 0.4

DNW-Baseline 0 srgb 39.1503 0.9521 1

LDResNet 0 srgb 39.114 0.9486 12

DRSR_v1.4 0 srgb 39.0941 0.9509 0.257

COLA-B 0 srgb 39.0711 0.949 1

Path-Restore 0 srgb 39.0047 0.9542 0.15

DNW-MNV1 0 srgb 38.9866 0.9513 1

WDnCNN+ 0 srgb 38.8695 0.9501 0.4

NLH 0 srgb 38.8532 0.9525 0

MPA-Net* 0 srgb 38.8522 0.947 1.9

GRDN 0 srgb 38.6992 0.9469 0

C-BSN_DND 0 srgb 38.6004 0.9414 0

UNet-D 0 srgb 38.585 0.9467 0.2

SDAP (S)(E) 0 srgb 38.5632 0.9402 1

AMSNet-SE 0 srgb 38.5237 0.9402 111

SS-BSN(self-ensemble) 0 srgb 38.4577 0.9397 0

C-BSN_SIDD 0 srgb 38.4502 0.9392 0

AMSNet-MSE 0 srgb 38.435 0.94 111

NAFNet-text 0 srgb 38.4198 0.9435 1
Pixel-shuffle Downsampling 0 srgb 38.4005 0.9452 0

AMSNet-E 0 srgb 38.3877 0.939 111

SDAP (S) 0 srgb 38.1785 0.9322 1

AP-BSN + R^3 0 srgb 38.0936 0.9368 0

CBDNet 0 srgb 38.0564 0.9421 0.4

DSSNet 0 srgb 38.0391 0.9358 0
TWSC 0 srgb 37.9643 0.9416 0

DnCNN+ 0 srgb 37.9018 0.943 0.05

BM3D G 0 srgb 37.8654 0.9342 3.47

SDAP (E) 0 srgb 37.8562 0.9366 1

Bitonic MX 0 srgb 37.8547 0.936 7.49

DNW-AMC 0 srgb 37.7682 0.9292 1

SS-BSN 0 srgb 37.7235 0.9283 0

blur_P3(denoising_4) 0 srgb 37.7099 0.9402 0

SDAP 0 srgb 37.7095 0.9278 1

FFDNet+ 0 srgb 37.6107 0.9415 0

AP-BSN 0 srgb 37.4601 0.9244 0

MCWNNM 0 srgb 37.379 0.9294 0

LBD 0 srgb 37.3233 0.9431 0
AMSNet-MS 0 srgb 37.2904 0.9196 111

C2N(SD)_DIDN 0 srgb 37.2808 0.9237 0

ULBD 0 srgb 37.2179 0.9341 0.01

Nonblind_MCD 0 srgb 37.1976 0.9361 0

MCD 0 srgb 37.12 0.9353 0
Unsupervised Denoising 0 srgb 37.1034 0.9441 0

128-DnCNN tensorflow 0 srgb 37.0343 0.9324 0.04
SCD 0 srgb 36.9453 0.9299 0
Noise2Variance(p=7) 0 srgb 36.9284 0.9281 0.43

Noise2Variance 0 srgb 36.9223 0.9284 0.43

Noise2Code 0 srgb 36.7977 0.9223 -1
BM3D (no G) 0 srgb 36.7885 0.912 3.43

LYMNet-R 0 srgb 36.5765 0.9246 0

CVF-SID (S^2) 0 srgb 36.5038 0.9233 -1

CVF-SID (S) 0 srgb 36.4912 0.9241 -1

KSVD 0 srgb 36.49 0.8978

CVF-SID (T) 0 srgb 36.3077 0.9232 -1

MPI 0 srgb 36.2436 0.9162 30
NAC 0 srgb 36.1987 0.9252 0

DPN 0 srgb 36.1183 0.9194 600
DRNet 0 srgb 35.8881 0.9351 0

GCBD 0 srgb 35.5796 0.9217
NC 0 srgb 35.434 0.8841 18.5

MPI_faster 0 srgb 35.3976 0.9094 7
NI 0 srgb 35.1125 0.8778 1.2

DenoiseNet 0 srgb 35.0802 0.868 0.33

WNNM 0 srgb 34.67 0.8646

FoE 0 srgb 34.62 0.8845

advect_run_sRGB_02 0 srgb 34.5311 0.8763 1

BM3D 0 srgb 34.51 0.8507

MLP 0 srgb 34.23 0.8331

NCSR 0 srgb 34.05 0.8351

TNRD 0 srgb 33.65 0.8306

EPLL 0 srgb 33.51 0.8244

test2 0 srgb 32.4898 0.8856 0
shuffle_local 0 srgb 32.4898 0.8856 0

Original Noisy 0 srgb 29.836 0.7018 0