介绍
这是一个针对广泛范围低光照图像增强的基线和工具箱。该仓库支持超过15个基准并能进行极高分辨率(最高可达4000x6000)的低光照增强。我们的方法Retinexformer 在NTIRE 2024低光照增强挑战中获得了第二名。如果您觉得这个仓库有用,请给我们一个星星⭐,并在您的研究中引用我们的论文。谢谢。
奖项
新闻
- 2024.08.07: 我们在这里分享了绘制标题图(柱状对比图)的代码。 欢迎在您的研究中使用:smile:
- 2024.07.03: 我们分享了更多对比基线方法的结果,以帮助您的研究。 欢迎从Google Drive或百度网盘下载:smile:
- 2024.07.01: 一个增强版本的Retinexformer已被ECCV 2024接收。代码将发布,敬请期待。🚀
- 2024.05.12: 基于我们Retinexformer框架和此仓库的RetinexMamba已发布。首个低光照增强的Mamba工作。感谢作者的努力。
- 2024.03.22: 我们发布了
分布式数据并行 (DDP)
和混合精度
训练策略,帮助您训练更大的模型。同时发布了自集成
测试策略,帮助您获得更好的结果。此外,我们还发布了一种自适应分割-测试
测试策略用于高分辨率(可达4000x6000)的低光照图像增强。欢迎使用。🚀 - 2024.03.21: 我们的方法Retinexformer和MST++(NTIRE 2022光谱重建挑战赛冠军)在NTIRE 2024低光照增强挑战赛中获得了前两名。代码、预训练权重、训练日志和增强结果已在此仓库中发布。欢迎使用!🚀
- 2024.02.15: NTIRE 2024低光照增强挑战赛开始。欢迎使用我们的Retinexformer或MST++(NTIRE 2022光谱重建挑战赛冠军)来参加此挑战赛!:trophy:
- 2023.11.03: 提供了KinD、LLFlow和最新扩散模型的测试设置以及在LOL上的相应结果。请注意,我们不建议使用此测试设置,因为它使用真实结果的均值来获得更好的结果。但是,如果您想跟随KinD、LLFlow和最近基于扩散的方法进行公平比较,选择此测试设置是您的选择。详情请参考
测试
部分。 - 2023.11.02: Retinexformer已被添加到Awesome-Transformer-Attention收藏中。 💫
- 2023.10.20: 提供了Params和FLOPS评估函数。欢迎检查和使用。
- 2023.10.12: Retinexformer已被添加到ICCV-2023-paper收藏中。🚀
- 2023.10.10: Retinexformer已被添加到low-level-vision-paper-record收藏中。 ⭐
- 2023.10.06: Retinexformer已被添加到awesome-low-light-image-enhancement集合中。 :tada:
- 2023.09.20: 发布了一些关于ExDark夜间物体检测的结果。
- 2023.09.20: 代码、模型、结果和训练日志已发布。欢迎使用。 ⭐
- 2023.07.14: 我们的论文已被ICCV 2023接收。代码和模型将发布。 :rocket:
结果
-
LOL-v1、LOL-v2-real、LOL-v2-synthetic、SID、SMID、SDSD-indoor、SDSD-outdoor和MIT Adobe FiveK数据集的结果可从百度网盘(代码:
cyh2
)或Google Drive下载。 -
结果在LOL-v1、LOL-v2-real和LOL-v2-synthetic数据集上,与KinD、LLFlow和最近的扩散模型在相同测试设置下的结果可以从Baidu Disk (密码:cyh2) 或Google Drive 下载。
-
在NTIRE 2024低光增强数据集上的结果可以从Baidu Disk (密码:cyh2) 或Google Drive 下载。
-
在LIME、NPE、MEF、DICM和VV数据集上的结果可以从Baidu Disk (密码:cyh2) 或Google Drive 下载。
-
在ExDark夜间目标检测上的结果可以从Baidu Disk (密码:cyh2) 或Google Drive 下载。请使用这个仓库 进行ExDark数据集上的实验。
-
一些对比基线方法的结果分享在Google Drive和Baidu Disk。
在LOL-v1、LOL-v2-real、LOL-v2-synthetic、SID、SMID、SDSD-in和SDSD-out上的表现:
在与KinD、LLFlow和扩散模型相同测试设置下的LOL表现:
指标 | LOL-v1 | LOL-v2-real | LOL-v2-synthetic |
---|---|---|---|
PSNR | 27.18 | 27.71 | 29.04 |
SSIM | 0.850 | 0.856 | 0.939 |
请注意,我们并不建议使用这种测试设置,因为它使用了真实值的均值来获得更好的结果。但是,如果您想遵循KinD、LLFlow和最近基于扩散的工作,这是您的选择。详情请参考“测试”部分。
在NTIRE 2024测试挑战中的表现:
方法 | Retinexformer | MST++ | Ensemble |
---|---|---|---|
PSNR | 24.61 | 24.59 | 25.30 |
SSIM | 0.85 | 0.85 | 0.85 |
请随时查看Codalab leaderboard。我们的方法排名第二。
在MIT Adobe FiveK上的表现:
在LIME、NPE、MEF、DICM和VV上的表现:
在ExDark夜间目标检测上的表现:
画廊
1. 创建环境
我们建议您使用pytorch 1.11重新实现我们ICCV 2023论文中的结果,并使用pytorch 2重新实现NTIRE 2024挑战中的结果,因为pytorch 2可以在混合精度训练中节省更多内存。
1.1 安装Pytorch 1.11的环境
- 创建Conda环境
conda create -n Retinexformer python=3.7
conda activate Retinexformer
- 安装依赖
conda install pytorch=1.11 torchvision cudatoolkit=11.3 -c pytorch
pip install matplotlib scikit-learn scikit-image opencv-python yacs joblib natsort h5py tqdm tensorboard
pip install einops gdown addict future lmdb numpy pyyaml requests scipy yapf lpips
- 安装BasicSR
python setup.py develop --no_cuda_ext
1.2 安装Pytorch 2的环境
- 创建Conda环境
conda create -n torch2 python=3.9 -y
conda activate torch2
- 安装依赖
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
pip install matplotlib scikit-learn scikit-image opencv-python yacs joblib natsort h5py tqdm tensorboard
pip install einops gdown addict future lmdb numpy pyyaml requests scipy yapf lpips thop timm
- 安装BasicSR
python setup.py develop --no_cuda_ext
2. 准备数据集
下载以下数据集:
LOL-v1 Baidu Disk (密码:cyh2), Google Drive
LOL-v2 Baidu Disk (密码:cyh2), Google Drive
SID Baidu Disk (密码:gplv), Google Drive
SMID Baidu Disk (密码:btux), Google Drive
SDSD-indoor Baidu Disk (密码:jo1v), Google Drive
SDSD-outdoor Baidu Disk (密码:uibk), Google Drive
MIT-Adobe FiveK Baidu Disk (密码: cyh2), Google Drive, 官方
NTIRE 2024 Baidu Disk (密码: cyh2), Google Drive链接分别为训练输入,训练GT和mini-val集。
注意:
(1) 请使用bandizip联合解压SMID、SDSD-indoor和SDSD-outdoor的.zip
和.z01
文件
(2) 请按照SRGB输出模式处理MIT Adobe FiveK数据集的原始图像,或者直接下载并使用我们处理过的SRGB图像对,链接在Baidu Disk (密码:cyh2) 和Google Drive。
请下载 text_list.txt
,可以从谷歌云盘 或者百度网盘 (密码: ggbh
)获取,然后将其放入 data/SMID/SMID_Long_np/
文件夹中。
然后将这些数据集组织如下:
|--data
| |--LOLv1
| | |--Train
| | | |--input
| | | | |--100.png
| | | | |--101.png
| | | | ...
| | | |--target
| | | | |--100.png
| | | | |--101.png
| | | | ...
| | |--Test
| | | |--input
| | | | |--111.png
| | | | |--146.png
| | | | ...
| | | |--target
| | | | |--111.png
| | | | |--146.png
| | | | ...
| |--LOLv2
| | |--Real_captured
| | | |--Train
| | | | |--Low
| | | | | |--00001.png
| | | | | |--00002.png
| | | | ...
| | | | |--Normal
| | | | | |--00001.png
| | | | | |--00002.png
| | | | ...
| | | |--Test
| | | | |--Low
| | | | | |--00690.png
| | | | | |--00691.png
| | | | ...
| | | | |--Normal
| | | | | |--00690.png
| | | | | |--00691.png
| | | | ...
| | |--Synthetic
| | | |--Train
| | | | |--Low
| | | | | |--r000da54ft.png
| | | | | |--r02e1abe2t.png
| | | | | ...
| | | | |--Normal
| | | | | |--r000da54ft.png
| | | | | |--r02e1abe2t.png
| | | | | ...
| | | |--Test
| | | | |--Low
| | | | | |--r00816405t.png
| | | | | |--r02189767t.png
| | | | | ...
| | | | |--Normal
| | | | | |--r00816405t.png
| | | | | |--r02189767t.png
| | | | | ...
| |--SDSD
| | |--indoor_static_np
| | | |--input
| | | | |--pair1
| | | | | |--0001.npy
| | | | | |--0002.npy
| | | | | ...
| | | | |--pair2
| | | | | |--0001.npy
| | | | | |--0002.npy
| | | | | ...
| | | | ...
| | | |--GT
| | | | |--pair1
| | | | | |--0001.npy
| | | | | |--0002.npy
| | | | | ...
| | | | |--pair2
| | | | | |--0001.npy
| | | | | |--0002.npy
| | | | | ...
| | | | ...
| | |--outdoor_static_np
| | | |--input
| | | | |--MVI_0898
| | | | | |--0001.npy
| | | | | |--0002.npy
| | | | | ...
| | | | |--MVI_0918
| | | | | |--0001.npy
| | | | | |--0002.npy
| | | | | ...
| | | | ...
| | | |--GT
| | | | |--MVI_0898
| | | | | |--0001.npy
| | | | | |--0002.npy
| | | | | ...
| | | | |--MVI_0918
| | | | | |--0001.npy
| | | | | |--0002.npy
| | | | | ...
| | | | ...
| |--SID
| | |--short_sid2
| | | |--00001
| | | | |--00001_00_0.04s.npy
| | | | |--00001_00_0.1s.npy
| | | | |--00001_01_0.04s.npy
| | | | |--00001_01_0.1s.npy
| | | ...
| | | |--00002
| | | | |--00002_00_0.04s.npy
| | | | |--00002_00_0.1s.npy
| | | | |--00002_01_0.04s.npy
| | | | |--00002_01_0.1s.npy
| | | ...
| | | ...
| | |--long_sid2
| | | |--00001
| | | | |--00001_00_0.04s.npy
| | | | |--00001_00_0.1s.npy
| | | | |--00001_01_0.04s.npy
| | | | |--00001_01_0.1s.npy
| | | ...
| | | |--00002
| | | | |--00002_00_0.04s.npy
| | | | |--00002_00_0.1s.npy
| | | | |--00002_01_0.04s.npy
| | | | |--00002_01_0.1s.npy
| | | ...
| | | ...
| |--SMID
| | |--SMID_LQ_np
| | | |--0001
| | | | |--0001.npy
| | | | |--0002.npy
| | | ...
| | | |--0002
| | | | |--0001.npy
| | | | |--0002.npy
| | | ...
| | | ...
| | |--SMID_Long_np
| | | |--text_list.txt
| | | |--0001
| | | | |--0001.npy
| | | | |--0002.npy
| | | ...
| | | |--0002
| | | | |--0001.npy
| | | | |--0002.npy
| | | ...
| | | ...
| |--FiveK
| | |--train
| | | |--input
| | | | |--a0099-kme_264.jpg
| | | | |--a0101-kme_610.jpg
| | | ...
| | | |--target
| | | | |--a0099-kme_264.jpg
| | | | |--a0101-kme_610.jpg
| | | ...
| | |--test
| | | |--input
| | | | |--a4574-DSC_0038.jpg
| | | | |--a4576-DSC_0217.jpg
| | | ...
| | | |--target
| | | | |--a4574-DSC_0038.jpg
| | | | |--a4576-DSC_0217.jpg
| | | ...
| |--NTIRE
| | |--train
| | | |--input
| | | | |--1.png
| | | | |--3.png
| | | ...
| | | |--target
| | | | |--1.png
| | | | |--3.png
| | | ...
| | |--minival
| | | |--input
| | | | |--1.png
| | | | |--31.png
| | | ...
| | | |--target
| | | | |--1.png
| | | | |--31.png
| | | ...
我们还提供了 LIME, NPE, MEF, DICM 和 VV 数据集的下载链接,这些数据集没有地面真值:
3. 测试
从百度网盘(密码: cyh2
)或者谷歌云盘下载我们的模型。将它们放入 pretrained_weights
文件夹。
# 激活环境
conda activate Retinexformer
# LOL-v1
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_LOL_v1.yml --weights pretrained_weights/LOL_v1.pth --dataset LOL_v1
# LOL-v2-real
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_LOL_v2_real.yml --weights pretrained_weights/LOL_v2_real.pth --dataset LOL_v2_real
# LOL-v2-synthetic
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_LOL_v2_synthetic.yml --weights pretrained_weights/LOL_v2_synthetic.pth --dataset LOL_v2_synthetic
# SID
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_SID.yml --weights pretrained_weights/SID.pth --dataset SID
# SMID
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_SMID.yml --weights pretrained_weights/SMID.pth --dataset SMID
# SDSD-indoor
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_SDSD_indoor.yml --weights pretrained_weights/SDSD_indoor.pth --dataset SDSD_indoor
# SDSD-outdoor
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_SDSD_outdoor.yml --weights pretrained_weights/SDSD_outdoor.pth --dataset SDSD_outdoor
# FiveK
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_FiveK.yml --weights pretrained_weights/FiveK.pth --dataset FiveK
# NTIRE
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_NTIRE.yml --weights pretrained_weights/NTIRE.pth --dataset NTIRE --self_ensemble
# MST_Plus_Plus trained with 4 GPUs on NTIRE
python3 Enhancement/test_from_dataset.py --opt Options/MST_Plus_Plus_NTIRE_4x1800.yml --weights pretrained_weights/MST_Plus_Plus_4x1800.pth --dataset NTIRE --self_ensemble
# MST_Plus_Plus trained with 8 GPUs on NTIRE
python3 Enhancement/test_from_dataset.py --opt Options/MST_Plus_Plus_NTIRE_8x1150.yml --weights pretrained_weights/MST_Plus_Plus_8x1150.pth --dataset NTIRE --self_ensemble
- #### Self-ensemble testing strategy
We add the self-ensemble strategy in the testing code to derive better results. Just add a `--self_ensemble` action at the end of the above test command to use it.
- #### The same test setting as LLFlow, KinD, and recent diffusion models
We provide the same test setting as LLFlow, KinD, and recent diffusion models. Please note that we do not suggest this test setting because it uses the mean of ground truth to enhance the output of the model. But if you want to follow this test setting, just add a `--GT_mean` action at the end of the above test command as
# LOL-v1
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_LOL_v1.yml --weights pretrained_weights/LOL_v1.pth --dataset LOL_v1 --GT_mean
# LOL-v2-real
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_LOL_v2_real.yml --weights pretrained_weights/LOL_v2_real.pth --dataset LOL_v2_real --GT_mean
# LOL-v2-synthetic
python3 Enhancement/test_from_dataset.py --opt Options/RetinexFormer_LOL_v2_synthetic.yml --weights pretrained_weights/LOL_v2_synthetic.pth --dataset LOL_v2_synthetic --GT_mean
- #### Evaluating the Params and FLOPS of models
We have provided a function `my_summary()` in `Enhancement/utils.py`, please use this function to evaluate the parameters and computational complexity of the models, especially the Transformers as
from utils import my_summary
my_summary(RetinexFormer(), 256, 256, 3, 1)
## 4. Training
Feel free to check our training logs from [Baidu Disk](https://pan.baidu.com/s/16NtLba_ANe3Vzji-eZ1xAA?pwd=cyh2) (code: `cyh2`) or [Google Drive](https://drive.google.com/drive/folders/1HU_wEn_95Hakxi_ze-pS6Htikmml5MTA?usp=sharing)
We suggest you use the environment with pytorch 2 to train our model on the NTIRE 2024 dataset and the environment with pytorch 1.11 to train our model on other datasets.
# activate the enviroment
conda activate Retinexformer
# LOL-v1
python3 basicsr/train.py --opt Options/RetinexFormer_LOL_v1.yml
# LOL-v2-real
python3 basicsr/train.py --opt Options/RetinexFormer_LOL_v2_real.yml
# LOL-v2-synthetic
python3 basicsr/train.py --opt Options/RetinexFormer_LOL_v2_synthetic.yml
# SID
python3 basicsr/train.py --opt Options/RetinexFormer_SID.yml
# SMID
python3 basicsr/train.py --opt Options/RetinexFormer_SMID.yml
# SDSD-indoor
python3 basicsr/train.py --opt Options/RetinexFormer_SDSD_indoor.yml
# SDSD-outdoor
python3 basicsr/train.py --opt Options/RetinexFormer_SDSD_outdoor.yml
# FiveK
python3 basicsr/train.py --opt Options/RetinexFormer_FiveK.yml
Train our Retinexformer and MST++ with the distributed data parallel (DDP) strategy of pytorch on the NTIRE 2024 Low-Light Enhancement dataset. Please note that we use the mix-precision strategy in the training process, which is controlled by the bool hyperparameter `use_amp` in the config file.
# activate the enviroment
conda activate torch2
# Train Retinexformer with 8 GPUs on NTIRE
bash train_multigpu.sh Options/RetinexFormer_NTIRE_8x2000.yml 0,1,2,3,4,5,6,7 4321
# Train MST++ with 4 GPUs on NTIRE
bash train_multigpu.sh Options/RetinexFormer_NTIRE_4x1800.yml 0,1,2,3,4,5,6,7 4329
# Train MST++ with 8 GPUs on NTIRE
bash train_multigpu.sh Options/MST_Plus_Plus_NTIRE_8x1150.yml 0,1,2,3,4,5,6,7 4343
## 5. Citation
@InProceedings{Cai_2023_ICCV,
author = {Cai, Yuanhao and Bian, Hao and Lin, Jing and Wang, Haoqian and Timofte, Radu and Zhang, Yulun},
title = {Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
pages = {12504-12513}
}
@inproceedings{retinexformer,
title={Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement},
author={Yuanhao Cai and Hao Bian and Jing Lin and Haoqian Wang and Radu Timofte and Yulun Zhang},
booktitle={ICCV},
year={2023}
}
# MST++
@inproceedings{mst,
title={Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction},
author={Yuanhao Cai and Jing Lin and Xiaowan Hu and Haoqian Wang and Xin Yuan and Yulun Zhang and Radu Timofte and Luc Van Gool},
booktitle={CVPR},
year={2022}
}