YangSijie/Report
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

add new notes

Browse Source
This commit is contained in:
杨思洁 2024-11-15 16:44:56 +08:00
parent 5ef8c9bd25
commit 6d48954367
9 changed files with 930 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
Docs/2024-10-25/.idea/.gitignore generated vendored Normal file
View File

@ -0,0 +1,8 @@
# Default ignored files
/shelf/
/workspace.xml
# Editor-based HTTP Client requests
/httpRequests/
# Datasource local storage ignored files
/dataSources/
/dataSources.local.xml

24
Docs/2024-10-25/.idea/inspectionProfiles/Project_Default.xml generated Normal file
View File

@ -0,0 +1,24 @@
<component name="InspectionProjectProfileManager">
<profile version="1.0">
<option name="myName" value="Project Default" />
<inspection_tool class="Eslint" enabled="true" level="WARNING" enabled_by_default="true" />
<inspection_tool class="PyCompatibilityInspection" enabled="true" level="WARNING" enabled_by_default="true">
<option name="ourVersions">
<value>
<list size="2">
<item index="0" class="java.lang.String" itemvalue="2.7" />
<item index="1" class="java.lang.String" itemvalue="3.11" />
</list>
</value>
</option>
</inspection_tool>
<inspection_tool class="PyPep8NamingInspection" enabled="true" level="WEAK WARNING" enabled_by_default="true">
<option name="ignoredErrors">
<list>
<option value="N803" />
<option value="N806" />
</list>
</option>
</inspection_tool>
</profile>
</component>

6
Docs/2024-10-25/.idea/inspectionProfiles/profiles_settings.xml generated Normal file
View File

@ -0,0 +1,6 @@
<component name="InspectionProjectProfileManager">
<settings>
<option name="USE_PROJECT_PROFILE" value="false" />
<version value="1.0" />
</settings>
</component>

7
Docs/2024-10-25/.idea/misc.xml generated Normal file
View File

@ -0,0 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="Black">
<option name="sdkName" value="Python 3.9" />
</component>
<component name="ProjectRootManager" version="2" project-jdk-name="Python 3.9" project-jdk-type="Python SDK" />
</project>

8
Docs/2024-10-25/.idea/modules.xml generated Normal file
View File

@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ProjectModuleManager">
<modules>
<module fileurl="file://$PROJECT_DIR$/.idea/2024-10-25.iml" filepath="$PROJECT_DIR$/.idea/2024-10-25.iml" />
</modules>
</component>
</project>

6
Docs/2024-10-25/.idea/vcs.xml generated Normal file
View File

@ -0,0 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="VcsDirectoryMappings">
<mapping directory="$PROJECT_DIR$/../.." vcs="Git" />
</component>
</project>

805
Docs/2024-11-15/2024CVPRPaper.md Normal file
View File

@ -0,0 +1,805 @@
# CVPR2024｜底层视觉相关论文汇总
CVPR2024底层视觉(Low-Level Vision)相关的论文和代码，包括超分辨率，图像去雨，图像去雾，去模糊，去噪，图像恢复，图像增强，图像去摩尔纹，图像修复，图像质量评价，插帧，图像/视频压缩等任务，具体如下。
https://zhuanlan.zhihu.com/p/684196283
CVPR2024官网：https://cvpr.thecvf.com/Conferences/2024
CVPR接收论文列表：https://cvpr.thecvf.com/Conferences/2024/AcceptedPapers
CVPR完整论文库：https://openaccess.thecvf.com/CVPR2024
开会时间：2024年6月17日-6月21日
论文接收公布时间：2024年2月27日
# 相关方法概览
1.超分辨率(Super-Resolution)
2.图像去雨(Image Deraining)
3.图像去雾(Image Dehazing)
4.去模糊(Deblurring)
5.去噪(Denoising)
6.图像恢复(Image Restoration)
7.图像增强(Image Enhancement)
8.图像修复(Inpainting)
9.高动态范围成像(HDR Imaging)
10.图像质量评价(Image Quality Assessment)
11.插帧(Frame Interpolation)
12.视频/图像压缩(Video/Image Compression)
13.压缩图像质量增强(Compressed Image Quality Enhancement)
14.图像去反光(Image Reflection Removal)
15.图像去阴影(Image Shadow Removal)
16.图像上色(Image Colorization)
17.图像和谐化(Image Harmonization)
18.视频稳相(Video Stabilization)
19.图像融合(Image Fusion)
20.其他任务(Others)
## 1.超分辨率(Super-Resolution)
**AdaBM: On-the-Fly Adaptive Bit Mapping for Image Super-Resolution**
* Paper: https://arxiv.org/abs/2404.03296
* Code: https://github.com/Cheeun/AdaBM
**A Dynamic Kernel Prior Model for Unsupervised Blind Image Super-Resolution**
* Paper: https://arxiv.org/abs/2404.15620
* Code: https://github.com/XYLGroup/DKP
**APISR: Anime Production Inspired Real-World Anime Super-Resolution**
* Paper: https://arxiv.org/abs/2403.01598
* Code: https://github.com/Kiteretsu77/APISR
**Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder**
* Paper: https://arxiv.org/abs/2403.10255v1
* Code: https://github.com/zhenshij/arbitrary-scale-diffusion
**Beyond Image Super-Resolution for Image Recognition with Task-Driven Perceptual Loss**
* Paper: https://arxiv.org/abs/2404.01692
* Code: https://github.com/JaehaKim97/SR4IR
**Bilateral Event Mining and Complementary for Event Stream Super-Resolution**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Huang_Bilateral_Event_Mining_and_Complementary_for_Event_Stream_Super-Resolution_CVPR_2024_paper.html
* Code: https://github.com/Lqm26/BMCNet-ESR
**Boosting Flow-based Generative Super-Resolution Models via Learned Prior**
* Paper: https://arxiv.org/abs/2403.10988
* Code: https://github.com/liyuantsao/FlowSR-LP
**Building Bridges across Spatial and Temporal Resolutions: Reference-Based Super-Resolution via Change Priors and Conditional Diffusion Model**
* Paper: https://arxiv.org/abs/2403.17460
* Code: https://github.com/dongrunmin/RefDiff
**CAMixerSR: Only Details Need More “Attention”**
* Paper: https://arxiv.org/abs/2402.19289
* Code: https://github.com/icandle/CAMixerSR
**CFAT: Unleashing Triangular Windows for Image Super-resolution**
* Paper: https://arxiv.org/abs/2403.16143
* Code: https://github.com/rayabhisek123/CFAT
**Continuous Optical Zooming: A Benchmark for Arbitrary-Scale Image Super-Resolution in Real World**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Fu_Continuous_Optical_Zooming_A_Benchmark_for_Arbitrary-Scale_Image_Super-Resolution_in_CVPR_2024_paper.html
* Code: https://github.com/pf0607/COZ
**CoSeR: Bridging Image and Language for Cognitive Super-Resolution**
* Paper: https://arxiv.org/abs/2311.16512
* Code: https://github.com/VINHYU/CoSeR
**CDFormer: When Degradation Prediction Embraces Diffusion Model for Blind Image Super-Resolution**
* Paper: https://arxiv.org/abs/2405.07648
* Code: https://github.com/I2-Multimedia-Lab/CDFormer
**CycleINR: Cycle Implicit Neural Representation for Arbitrary-Scale Volumetric Super-Resolution of Medical Data**
* Paper: https://arxiv.org/abs/2404.04878
* Code:
**Diffusion-based Blind Text Image Super-Resolution**
* Paper: https://arxiv.org/abs/2312.08886
* Code: https://github.com/YuzheZhang-1999/DiffTSR
**DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF**
* Paper: https://arxiv.org/abs/2404.00874
* Code:
**Image Processing GNN: Breaking Rigidity in Super-Resolution**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Tian_Image_Processing_GNN_Breaking_Rigidity_in_Super-Resolution_CVPR_2024_paper.html
* Code: https://github.com/huawei-noah/Efficient-Computing/tree/master/LowLevel/IPG
**Latent Modulated Function for Computational Optimal Continuous Image Representation**
* Paper: https://arxiv.org/abs/2404.16451
* Code: https://github.com/HeZongyao/LMF
**Learning Coupled Dictionaries from Unpaired Data for Image Super-Resolution**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Wang_Learning_Coupled_Dictionaries_from_Unpaired_Data_for_Image_Super-Resolution_CVPR_2024_paper.html
* Code:
**Learning Large-Factor EM Image Super-Resolution with Generative Priors**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shou_Learning_Large-Factor_EM_Image_Super-Resolution_with_Generative_Priors_CVPR_2024_paper.html
* Code: https://github.com/jtshou/GPEMSR
**Low-Res Leads the Way: Improving Generalization for Super-Resolution by Self-Supervised Learning**
* Paper: https://arxiv.org/abs/2403.02601
* Code: https://github.com/haoyuc/LWay
**Navigating Beyond Dropout: An Intriguing Solution towards Generalizable Image Super-Resolution**
* Paper: https://arxiv.org/abs/2402.18929v2
* Code: https://github.com/Dreamzz5/Simple-Align
**Neural Super-Resolution for Real-time Rendering with Radiance Demodulation**
* Paper: https://arxiv.org/abs/2308.06699
* Code: https://github.com/Riga2/NSRD
**Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution**
* Paper: https://arxiv.org/abs/2404.04785
* Code: https://github.com/GuangYuanKK/DiffMSR
**SeD: Semantic-Aware Discriminator for Image Super-Resolution**
* Paper: https://arxiv.org/abs/2402.19387
* Code: https://github.com/lbc12345/SeD
**SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution**
* Paper: https://arxiv.org/abs/2311.16518
* Code: https://github.com/cswry/SeeSR
**Self-Adaptive Reality-Guided Diffusion for Artifact-Free Super-Resolution**
* Paper: https://arxiv.org/abs/2403.16643
* Code: https://github.com/ProAirVerse/Self-Adaptive-Guidance-Diffusion
**SinSR: Diffusion-Based Image Super-Resolution in a Single Step**
* Paper: https://github.com/wyf0912/SinSR/blob/main/main.pdf
* Code: https://github.com/wyf0912/SinSR
**Super-Resolution Reconstruction from Bayer-Pattern Spike Streams**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Dong_Super-Resolution_Reconstruction_from_Bayer-Pattern_Spike_Streams_CVPR_2024_paper.html
* Code: https://github.com/csycdong/CSCSR
**Text-guided Explorable Image Super-resolution**
* Paper: https://arxiv.org/abs/2403.01124
* Code:
**Training Generative Image Super-Resolution Models by Wavelet-Domain Losses Enables Better Control of Artifacts**
* Paper: https://arxiv.org/abs/2402.19215
* Code: https://github.com/mandalinadagi/wgsr
**Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary**
* Paper: https://arxiv.org/abs/2401.08209
* Code: https://github.com/LabShuHangGU/Adaptive-Token-Dictionary
**Uncertainty-Aware Source-Free Adaptive Image Super-Resolution with Wavelet Augmentation Transformer**
* Paper: https://arxiv.org/abs/2303.17783
* Code:
**Universal Robustness via Median Randomized Smoothing for Real-World Super-Resolution**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chaouai_Universal_Robustness_via_Median_Randomized_Smoothing_for_Real-World_Super-Resolution_CVPR_2024_paper.html
* Code:
### Video Super-Resolution
**Enhancing Video Super-Resolution via Implicit Resampling-based Alignment**
* Paper: https://github.com/kai422/IART/blob/main/arxiv.pdf
* Code: https://github.com/kai422/IART
**FMA-Net: Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring**
* Paper: https://arxiv.org/abs/2401.03707
* Code: https://github.com/KAIST-VICLab/FMA-Net
**Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution**
* Paper: https://arxiv.org/abs/2403.17000
* Code:
**Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution**
* Paper: https://arxiv.org/abs/2312.06640
* Code: https://github.com/sczhou/Upscale-A-Video
**Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention**
* Paper: https://arxiv.org/abs/2401.06312
* Code: https://github.com/LabShuHangGU/MIA-VSR
## 2.图像去雨(Image Deraining)
**Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining**
* Paper: https://arxiv.org/abs/2404.01547
* Code: https://github.com/cschenxiang/NeRD-Rain
## 3.图像去雾(Image Dehazing)
**A Semi-supervised Nighttime Dehazing Baseline with Spatial-Frequency Aware and Realistic Brightness Constraint**
* Paper: https://arxiv.org/abs/2403.18548
* Code: https://github.com/Xiaofeng-life/SFSNiD
**Depth Information Assisted Collaborative Mutual Promotion Network for Single Image Dehazing**
* Paper: https://arxiv.org/abs/2403.01105
* Code:
**ODCR: Orthogonal Decoupling Contrastive Regularization for Unpaired Image Dehazing**
* Paper: https://arxiv.org/abs/2404.17825v1
* Code:
### Video Dehazing
**Driving-Video Dehazing with Non-Aligned Regularization for Safety Assistance**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Fan_Driving-Video_Dehazing_with_Non-Aligned_Regularization_for_Safety_Assistance_CVPR_2024_paper.html
* Code:
## 4.去模糊(Deblurring)
**A Unified Framework for Microscopy Defocus Deblur with Multi-Pyramid Transformer and Contrastive Learning**
* Paper: https://arxiv.org/abs/2403.02611
* Code: https://github.com/PieceZhang/MPT-CataBlur
**AdaRevD: Adaptive Patch Exiting Reversible Decoder Pushes the Limit of Image Deblurring**
* Paper: https://github.com/INVOKERer/AdaRevD/blob/master/AdaRevD.pdf
* Code: https://github.com/INVOKERer/AdaRevD
**Blur2Blur: Blur Conversion for Unsupervised Image Deblurring on Unknown Domains**
* Paper: https://arxiv.org/abs/2403.16205
* Code: https://github.com/VinAIResearch/Blur2Blur
**Fourier Priors-Guided Diffusion for Zero-Shot Joint Low-Light Enhancement and Deblurring**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Lv_Fourier_Priors-Guided_Diffusion_for_Zero-Shot_Joint_Low-Light_Enhancement_and_Deblurring_CVPR_2024_paper.html
* Code:
**ID-Blau: Image Deblurring by Implicit Diffusion-based reBLurring AUgmentation**
* Paper：https://arxiv.org/abs/2312.10998
* Code: https://github.com/plusgood-steven/ID-Blau
**LDP: Language-driven Dual-Pixel Image Defocus Deblurring Network**
* Paper: https://arxiv.org/abs/2307.09815
* Code: https://github.com/noxsine/LDP
**Mitigating Motion Blur in Neural Radiance Fields with Events and Frames**
* Paper: https://rpg.ifi.uzh.ch/docs/CVPR24_Cannici.pdf
* Code: https://github.com/uzh-rpg/EvDeblurNeRF
**Motion-adaptive Separable Collaborative Filters for Blind Motion Deblurring**
* Paper: https://arxiv.org/abs/2404.13153
* Code: https://github.com/ChengxuLiu/MISCFilter
**Motion Blur Decomposition with Cross-shutter Guidance**
* Paper: https://arxiv.org/abs/2404.01120
* Code: https://github.com/jixiang2016/dualBR
**Real-World Efficient Blind Motion Deblurring via Blur Pixel Discretization**
* Paper: https://arxiv.org/abs/2404.12168
* Code:
**Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Spike-guided_Motion_Deblurring_with_Unknown_Modal_Spatiotemporal_Alignment_CVPR_2024_paper.html
* Code: https://github.com/Leozhangjiyuan/UaSDN
**Unsupervised Blind Image Deblurring Based on Self-Enhancement**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chen_Unsupervised_Blind_Image_Deblurring_Based_on_Self-Enhancement_CVPR_2024_paper.html
* Code:
### Video Deblurring
**Blur-aware Spatio-temporal Sparse Transformer for Video Deblurring**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Blur-aware_Spatio-temporal_Sparse_Transformer_for_Video_Deblurring_CVPR_2024_paper.html
* Code: https://github.com/huicongzhang/BSSTNet
**EVS-assisted Joint Deblurring Rolling-Shutter Correction and Video Frame Interpolation through Sensor Inverse Modeling**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Jiang_EVS-assisted_Joint_Deblurring_Rolling-Shutter_Correction_and_Video_Frame_Interpolation_through_CVPR_2024_paper.html
* Code:
**Frequency-aware Event-based Video Deblurring for Real-World Motion Blur**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Kim_Frequency-aware_Event-based_Video_Deblurring_for_Real-World_Motion_Blur_CVPR_2024_paper.html
* Code:
**Latency Correction for Event-guided Deblurring and Frame Interpolation**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Yang_Latency_Correction_for_Event-guided_Deblurring_and_Frame_Interpolation_CVPR_2024_paper.html
****Code:
## 5.去噪(Denoising)
**LAN: Learning to Adapt Noise for Image Denoising**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Kim_LAN_Learning_to_Adapt_Noise_for_Image_Denoising_CVPR_2024_paper.html
* Code:
**LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising**
* Paper: https://arxiv.org/abs/2405.19718
* Code:
**Robust Image Denoising through Adversarial Frequency Mixup**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Ryou_Robust_Image_Denoising_through_Adversarial_Frequency_Mixup_CVPR_2024_paper.html
* Code: https://github.com/dhryougit/AFM
**Real-World Mobile Image Denoising Dataset with Efficient Baselines**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Flepp_Real-World_Mobile_Image_Denoising_Dataset_with_Efficient_Baselines_CVPR_2024_paper.html
* Code:
**SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder**
* Paper: https://arxiv.org/abs/2403.17502
* Code: https://github.com/zhengdharia/SeNM-VAE
**Transfer CLIP for Generalizable Image Denoising**
* Paper: https://arxiv.org/abs/2403.15132
* Code:
**Unmixing Diffusion for Self-Supervised Hyperspectral Image Denoising**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zeng_Unmixing_Diffusion_for_Self-Supervised_Hyperspectral_Image_Denoising_CVPR_2024_paper.html
* Code:
**ZERO-IG: Zero-Shot Illumination-Guided Joint Denoising and Adaptive Enhancement for Low-Light Images**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shi_ZERO-IG_Zero-Shot_Illumination-Guided_Joint_Denoising_and_Adaptive_Enhancement_for_Low-Light_CVPR_2024_paper.html
* Code: https://github.com/Doyle59217/ZeroIG
## 6.图像恢复(Image Restoration)
**Adapt or Perish: Adaptive Sparse Transformer with Attentive Feature Refinement for Image Restoration**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhou_Adapt_or_Perish_Adaptive_Sparse_Transformer_with_Attentive_Feature_Refinement_CVPR_2024_paper.html
* Code: https://github.com/joshyZhou/AST
**Boosting Image Restoration via Priors from Pre-trained Models**
* Paper: https://arxiv.org/abs/2403.06793
* Code:
**CoDe: An Explicit Content Decoupling Framework for Image Restoration**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Gu_CoDe_An_Explicit_Content_Decoupling_Framework_for_Image_Restoration_CVPR_2024_paper.html
* Code:
**Deep Equilibrium Diffusion Restoration with Parallel Sampling**
* Paper: https://arxiv.org/abs/2311.11600
* Code: https://github.com/caojiezhang/DeqIR
**Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks**
* Paper: https://arxiv.org/abs/2403.00644
* Code: https://github.com/yuhaoliu7456/Diff-Plugin
**Distilling Semantic Priors from SAM to Efficient Image Restoration Models**
* Paper: https://arxiv.org/abs/2403.16368
* Code:
**DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks**
* Paper: https://arxiv.org/abs/2405.04408
* Code: https://github.com/ZZZHANG-jx/DocRes
**HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models**
* Paper: https://arxiv.org/abs/2402.15865
* Code: https://github.com/LiPang/HIRDiff
**Image Restoration by Denoising Diffusion Models With Iteratively Preconditioned Guidance**
* Paper: https://arxiv.org/abs/2312.16519
* Code: https://github.com/tirer-lab/DDPG
**Improving Image Restoration through Removing Degradations in Textual Representations**
* Paper: https://arxiv.org/abs/2312.17334
* Code: https://github.com/mrluin/TextualDegRemoval
**Learning Degradation-unaware Representation with Prior-based Latent Transformations for Blind Face Restoration**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Xie_Learning_Degradation-unaware_Representation_with_Prior-based_Latent_Transformations_for_Blind_Face_CVPR_2024_paper.html
* Code:
**Learning Diffusion Texture Priors for Image Restoration**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Ye_Learning_Diffusion_Texture_Priors_for_Image_Restoration_CVPR_2024_paper.html
* Code:
**Look-Up Table Compression for Efficient Image Restoration**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Li_Look-Up_Table_Compression_for_Efficient_Image_Restoration_CVPR_2024_paper.html
* Code:
**Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration**
* Paper: https://arxiv.org/abs/2312.02918
* Code:
**PFStorer: Personalized Face Restoration and Super-Resolution**
* Paper: https://arxiv.org/abs/2403.08436
* Code:
**Restoration by Generation with Constrained Priors**
* Paper: https://arxiv.org/abs/2312.17161
* Code:
**Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild**
* Paper: https://arxiv.org/abs/2401.13627
* Code: https://github.com/Fanghua-Yu/SUPIR
**Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Model**
* Paper: https://arxiv.org/abs/2403.11157
* Code: https://github.com/iSEE-Laboratory/DiffUIR
**Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence**
* Paper: https://arxiv.org/abs/2404.13605
Code: https://github.com/Riponcs/Turb-Seg-Res
**WaveFace: Authentic Face Restoration with Efficient Frequency Recovery**
* Paper: https://arxiv.org/abs/2403.12760
* Code:
**Wavelet-based Fourier Information Interaction with Frequency Diffusion Adjustment for Underwater Image Restoration**
* Paper: https://arxiv.org/abs/2311.16845
* Code: https://github.com/zhihefang/wf-diff
## 7.图像增强(Image Enhancement)
**Color Shift Estimation-and-Correction for Image Enhancement**
* Paper: https://drive.google.com/file/d/1jZB2rW_I2WLTE5yNA4IZq9wb5p4NNOCR/view
* Code: https://github.com/yiyulics/CSEC
**Empowering Resampling Operation for Ultra-High-Definition Image Enhancement with Model-Aware Guidance**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Yu_Empowering_Resampling_Operation_for_Ultra-High-Definition_Image_Enhancement_with_Model-Aware_Guidance_CVPR_2024_paper.html
* Code: https://github.com/YPatrickW/LMAR
**Fourier Priors-Guided Diffusion for Zero-Shot Joint Low-Light Enhancement and Deblurring**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Lv_Fourier_Priors-Guided_Diffusion_for_Zero-Shot_Joint_Low-Light_Enhancement_and_Deblurring_CVPR_2024_paper.html
* Code:
**FlowIE：Efficient Image Enhancement via Rectified Flow**
* Paper: https://arxiv.org/abs/2406.00508
* Code: https://github.com/EternalEvan/FlowIE
**Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving**
* Paper: https://arxiv.org/abs/2404.04804
* Code: https://github.com/jinlong17/LightDiff
**Robust Depth Enhancement via Polarization Prompt Fusion Tuning**
* Paper: https://arxiv.org/abs/2404.04318
* Code: https://github.com/lastbasket/Polarization-Prompt-Fusion-Tuning
**Specularity Factorization for Low Light Enhancement**
* Paper: https://arxiv.org/abs/2404.01998
* Code:
**Towards Robust Event-guided Low-Light Image Enhancement: A Large-Scale Real-World Event-Image Dataset and Novel Approach**
* Paper: https://arxiv.org/abs/2404.00834
* Code: https://github.com/EthanLiang99/EvLight
**ZERO-IG: Zero-Shot Illumination-Guided Joint Denoising and Adaptive Enhancement for Low-Light Images**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shi_ZERO-IG_Zero-Shot_Illumination-Guided_Joint_Denoising_and_Adaptive_Enhancement_for_Low-Light_CVPR_2024_paper.html
* Code: https://github.com/Doyle59217/ZeroIG
**Zero-Reference Low-Light Enhancement via Physical Quadruple Priors**
* Paper: https://arxiv.org/abs/2403.12933
* Code: https://github.com/daooshee/QuadPrior
### Video Enhancement
**Binarized Low-light Raw Video Enhancement**
* Paper: https://arxiv.org/abs/2403.19944
* Code: https://github.com/zhanggengchen/BRVE
**UVEB: A Large-scale Benchmark and Baseline Towards Real-World Underwater Video Enhancement**
* Paper: https://arxiv.org/abs/2404.14542
* Code: https://github.com/yzbouc/UVEB
## 8.图像修复(Inpainting)
**Amodal Completion via Progressive Mixed Context Diffusion**
* Paper: https://arxiv.org/abs/2312.15540
* Code: https://github.com/k8xu/amodal
**Brush2Prompt: Contextual Prompt Generator for Object Inpainting**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chiu_Brush2Prompt_Contextual_Prompt_Generator_for_Object_Inpainting_CVPR_2024_paper.html
* Code:
**Don’t Look into the Dark: Latent Codes for Pluralistic Image Inpainting**
* Paper: https://arxiv.org/abs/2403.18186
* Code:
**Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting**
* Paper: https://arxiv.org/abs/2403.19898
* Code: https://github.com/htyjers/StrDiffusion
### Video Inpainting
**AVID: Any-Length Video Inpainting with Diffusion Model**
* Paper: https://arxiv.org/abs/2312.03816
* Code: https://github.com/zhang-zx/AVID
**Towards Language-Driven Video Inpainting via Multimodal Large Language Models**
* Paper: https://arxiv.org/abs/2401.10226
* Code: https://github.com/jianzongwu/Language-Driven-Video-Inpainting
## 9.高动态范围成像(HDR Imaging)
**CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment**
* Paper: https://arxiv.org/abs/2404.01123
* Code: https://github.com/hmin970922/CLIPtone/
**Deep Video Inverse Tone Mapping Based on Temporal Clues**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Ye_Deep_Video_Inverse_Tone_Mapping_Based_on_Temporal_Clues_CVPR_2024_paper.html
* Code: https://github.com/ye3why/VITM-TC
**Generating Content for HDR Deghosting from Frequency View**
* Paper: https://arxiv.org/abs/2404.00849
* Code:
**HDRFlow: Real-Time HDR Video Reconstruction with Large Motions**
* Paper: https://arxiv.org/abs/2403.03447
* Code: https://github.com/OpenImagingLab/HDRFlow
**Perceptual Assessment and Optimization of HDR Image Rendering**
* Paper: https://arxiv.org/abs/2310.12877v4
* Code: https://github.com/cpb68/HDRQA/
**Towards HDR and HFR Video from Rolling-Mixed-Bit Spikings**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chang_Towards_HDR_and_HFR_Video_from_Rolling-Mixed-Bit_Spikings_CVPR_2024_paper.html
* Code:
**Towards Real-World HDR Video Reconstruction: A Large-Scale Benchmark Dataset and A Two-Stage Alignment Network**
* Paper: https://arxiv.org/abs/2405.00244
* Code: https://github.com/yungsyu99/Real-HDRV
**Zero-Shot Structure-Preserving Diffusion Model for High Dynamic Range Tone Mapping**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhu_Zero-Shot_Structure-Preserving_Diffusion_Model_for_High_Dynamic_Range_Tone_Mapping_CVPR_2024_paper.html
* Code:
## 10.图像质量评价(Image Quality Assessment)
**Blind Image Quality Assessment Based on Geometric Order Learning**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shin_Blind_Image_Quality_Assessment_Based_on_Geometric_Order_Learning_CVPR_2024_paper.html
* Code: https://github.com/nhshin-mcl/QCN
**Boosting Image Quality Assessment through Efficient Transformer Adaptation with Local Feature Enhancement**
* Paper: https://arxiv.org/abs/2308.12001
* Code:
**Bridging the Synthetic-to-Authentic Gap: Distortion-Guided Unsupervised Domain Adaptation for Blind Image Quality Assessment**
* Paper: https://arxiv.org/abs/2405.04167
* Code:
**CLIB-FIQA: Face Image Quality Assessment with Confidence Calibration**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Ou_CLIB-FIQA_Face_Image_Quality_Assessment_with_Confidence_Calibration_CVPR_2024_paper.html
* Code:
**Contrastive Pre-Training with Multi-View Fusion for No-Reference Point Cloud Quality Assessment**
* Paper: https://arxiv.org/abs/2403.10066
* Code:
**Deep Generative Model based Rate-Distortion for Image Downscaling Assessment**
* Paper: https://arxiv.org/abs/2403.15139
* Code: https://github.com/Byronliang8/IDA-RD
**Defense Against Adversarial Attacks on No-Reference Image Quality Models with Gradient Norm Regularization**
* Paper: https://arxiv.org/abs/2403.11397
* Code: https://github.com/YangiD/DefenseIQA-NT
**DSL-FIQA: Assessing Facial Image Quality via Dual-Set Degradation Learning and Landmark-Guided Transformer**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Chen_DSL-FIQA_Assessing_Facial_Image_Quality_via_Dual-Set_Degradation_Learning_and_CVPR_2024_paper.html
* Code:
**EvalCrafter: Benchmarking and Evaluating Large Video Generation Models**
* Paper: https://arxiv.org/abs/2310.11440
* Code: https://github.com/evalcrafter/EvalCrafter
**FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment**
* Paper: https://arxiv.org/abs/2405.06887
* Code: https://github.com/PKU-ICST-MIPL/FineParser_CVPR2024
**KVQ: Kwai Video Quality Assessment for Short-form Videos**
* Paper: https://arxiv.org/abs/2402.07220
* Code: https://github.com/lixinustc/KVQ-Challenge-CVPR-NTIRE2024
**Learned Scanpaths Aid Blind Panoramic Video Quality Assessment**
* Paper: https://arxiv.org/abs/2404.00252
* Code: https://github.com/kalofan/AutoScanpathQA
**Modular Blind Video Quality Assessment**
* Paper: https://arxiv.org/abs/2402.19276
* Code: https://github.com/winwinwenwen77/ModularBVQA
**On the Content Bias in Fréchet Video Distance**
* Paper: https://arxiv.org/abs/2404.12391
* Code: https://github.com/songweige/content-debiased-fvd
**PTM-VQA: Efficient Video Quality Assessment Leveraging Diverse PreTrained Models from the Wild**
* Paper: https://arxiv.org/abs/2405.17765
* Code:
**Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models**
* Paper: https://arxiv.org/abs/2311.06783
* Code: https://github.com/Q-Future/Q-Instruct
## 11.插帧(Frame Interpolation)
**Data-Efficient Unsupervised Interpolation Without Any Intermediate Frame for 4D Medical Images**
Paper: https://arxiv.org/abs/2404.01464
Code: https://github.com/jungeun122333/UVI-Net
**IQ-VFI: Implicit Quadratic Motion Estimation for Video Frame Interpolation**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Hu_IQ-VFI_Implicit_Quadratic_Motion_Estimation_for_Video_Frame_Interpolation_CVPR_2024_paper.html
* Code:
**Perceptual-Oriented Video Frame Interpolation Via Asymmetric Synergistic Blending**
* Paper: https://arxiv.org/abs/2404.06692
* Code:
**Sparse Global Matching for Video Frame Interpolation with Large Motion**
* Paper: https://arxiv.org/abs/2404.06913
* Code: https://github.com/MCG-NJU/SGM-VFI
**SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation**
* Paper: https://arxiv.org/abs/2308.16876
* Code: https://github.com/neu-vi/SportsSloMo
**TTA-EVF: Test-Time Adaptation for Event-based Video Frame Interpolation via Reliable Pixel and Sample Estimation**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Cho_TTA-EVF_Test-Time_Adaptation_for_Event-based_Video_Frame_Interpolation_via_Reliable_CVPR_2024_paper.html
* Code: https://github.com/Chohoonhee/TTA-EVF
**Video Frame Interpolation via Direct Synthesis with the Event-based Reference**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Liu_Video_Frame_Interpolation_via_Direct_Synthesis_with_the_Event-based_Reference_CVPR_2024_paper.html
* Code:
**Video Interpolation with Diffusion Models**
* Paper: https://arxiv.org/abs/2404.01203
* Code:
## 12.视频/图像压缩(Video/Image Compression)
**C3: High-performance and low-complexity neural compression from a single image or video**
* Paper: https://arxiv.org/abs/2312.02753
* Code: https://github.com/google-deepmind/c3_neural_compression
**Generative Latent Coding for Ultra-Low Bitrate Image Compression**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Jia_Generative_Latent_Coding_for_Ultra-Low_Bitrate_Image_Compression_CVPR_2024_paper.html
* Code:
**Laplacian-guided Entropy Model in Neural Codec with Blur-dissipated Synthesis**
* Paper: https://arxiv.org/abs/2403.16258
* Code:
**Learned Lossless Image Compression based on Bit Plane Slicing**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Learned_Lossless_Image_Compression_based_on_Bit_Plane_Slicing_CVPR_2024_paper.html
* Code: https://github.com/ZZ022/ArIB-BPS
**Towards Backward-Compatible Continual Learning of Image Compression**
* Paper: https://arxiv.org/abs/2402.18862
* Code: https://gitlab.com/viper-purdue/continual-compression
### Video Compression
**Task-Aware Encoder Control for Deep Video Compression**
* Paper: https://arxiv.org/abs/2404.04848
* Code:
**Low-Latency Neural Stereo Streaming**
* Paper: https://arxiv.org/abs/2403.17879
* Code:
**Neural Video Compression with Feature Modulation**
* Paper: https://arxiv.org/abs/2402.17414
* Code: https://github.com/microsoft/DCVC
## 13.压缩图像质量增强(Compressed Image Quality Enhancement)
**CPGA: Coding Priors-Guided Aggregation Network for Compressed Video Quality Enhancement**
* Paper: https://arxiv.org/abs/2403.10362
* Code:
**Enhancing Quality of Compressed Images by Mitigating Enhancement Bias Towards Compression Domain**
* Paper: https://arxiv.org/abs/2402.17200
* Code:
## 14.图像去反光(Image Reflection Removal)
**Language-guided Image Reflection Separation**
* Paper: https://arxiv.org/abs/2402.11874
* Code:
**Revisiting Singlelmage Reflection Removal in the Wild**
* Paper: https://arxiv.org/abs/2311.17320
* Code: https://github.com/zhuyr97/Reflection_RemoVal_CVPR2024
## 15.图像去阴影(Image Shadow Removal)
**HomoFormer: Homogenized Transformer for Image Shadow Removal**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Xiao_HomoFormer_Homogenized_Transformer_for_Image_Shadow_Removal_CVPR_2024_paper.html
* Code: https://github.com/jiexiaou/HomoFormer
## 16.图像上色(Image Colorization)
**Automatic Controllable Colorization by Imagination**
* Paper: https://arxiv.org/abs/2404.05661
* Code: https://github.com/xy-cong/imagine-colorization
**Generative Quanta Color Imaging**
* Paper: https://arxiv.org/abs/2403.19066
* Code:
**Learning Inclusion Matching for Animation Paint Bucket Colorization**
* Paper: https://arxiv.org/abs/2403.18342
* Code: https://github.com/ykdai/BasicPBC
## 17.图像和谐化(Image Harmonization)
**Relightful Harmonization: Lighting-aware Portrait Background Replacement**
* Paper: https://arxiv.org/abs/2312.06886
* Code:
**Video Harmonization with Triplet Spatio-Temporal Variation Patterns**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Guo_Video_Harmonization_with_Triplet_Spatio-Temporal_Variation_Patterns_CVPR_2024_paper.html
* Code: https://github.com/zhenglab/VideoTripletTransformer
## 18.视频稳相(Video Stabilization)
**3D Multi-frame Fusion for Video Stabilization**
* Paper: https://arxiv.org/abs/2404.12887
* Code:
**Harnessing Meta-Learning for Improving Full-Frame Video Stabilization**
* Paper: https://arxiv.org/abs/2403.03662
* Code: https://github.com/MKashifAli/MetaVideoStab
## 19.图像融合(Image Fusion)
**Equivariant Multi-Modality Image Fusion**
* Paper： https://arxiv.org/abs/2305.11443
* Code: https://github.com/Zhaozixiang1228/MMIF-EMMA
**MRFS: Mutually Reinforcing Image Fusion and Segmentation**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_MRFS_Mutually_Reinforcing_Image_Fusion_and_Segmentation_CVPR_2024_paper.html
* Code: https://github.com/HaoZhang1018/MRFS
**Neural Spline Fields for Burst Image Fusion and Layer Separation**
* Paper: https://arxiv.org/abs/2312.14235
* Code: https://github.com/princeton-computational-imaging/NSF
**Probing Synergistic High-Order Interaction in Infrared and Visible Image Fusion**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zheng_Probing_Synergistic_High-Order_Interaction_in_Infrared_and_Visible_Image_Fusion_CVPR_2024_paper.html
* Code:
**Revisiting Spatial-Frequency Information Integration from a Hierarchical Perspective for Panchromatic and Multi-Spectral Image Fusion**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zheng_Probing_Synergistic_High-Order_Interaction_in_Infrared_and_Visible_Image_Fusion_CVPR_2024_paper.html
* Code:
**Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion**
* Paper: https://arxiv.org/abs/2403.16387
* Code: https://github.com/XunpengYi/Text-IF
**Task-Customized Mixture of Adapters for General Image Fusion**
* Paper: https://arxiv.org/abs/2403.12494
* Code: https://github.com/YangSun22/TC-MoA
## 20.其他任务(Others)
**Close Imitation of Expert Retouching for Black-and-White Photography**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Shin_Close_Imitation_of_Expert_Retouching_for_Black-and-White_Photography_CVPR_2024_paper.html
* Code: https://github.com/seunghyuns98/Decolorization
**Content-Adaptive Non-Local Convolution for Remote Sensing Pansharpening**
Paper: https://arxiv.org/abs/2404.07543
Code: https://github.com/Duanyll/CANConv
**DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model**
* Paper: https://arxiv.org/abs/2311.11417
* Code: https://github.com/PAN083/DiffSCI
**Dual Prior Unfolding for Snapshot Compressive Imaging**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Dual_Prior_Unfolding_for_Snapshot_Compressive_Imaging_CVPR_2024_paper.html
* Code: https://github.com/ZhangJC-2k/DPU
**Dual-Camera Smooth Zoom on Mobile Phones**
* Paper: https://arxiv.org/abs/2404.04908
* Code: https://github.com/ZcsrenlongZ/ZoomGS
**Dual-scale Transformer for Large-scale Single-Pixel Imaging**
* Paper: https://arxiv.org/abs/2404.05001
* Code: https://github.com/Gang-Qu/HATNet-SPI
**Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal**
* Paper: https://arxiv.org/abs/2403.07684
* Code: https://github.com/scott-yjyang/DiffTTA
**Language-driven All-in-one Adverse Weather Removal**
* Paper: https://arxiv.org/abs/2312.01381
* Code:
**Learning to Remove Wrinkled Transparent Film with Polarized Prior**
* Paper: https://arxiv.org/abs/2403.04368
* Code: https://github.com/jqtangust/FilmRemovalww
**Misalignment-Robust Frequency Distribution Loss for Image Transformation**
* Paper: https://arxiv.org/abs/2402.18192
* Code: https://github.com/eezkni/FDL
**On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation**
* Paper: https://arxiv.org/abs/2404.08540
* Code: https://github.com/agneet42/lang_depth
**ParamISP: Learned Forward and Inverse ISPs using Camera Parameters**
* Paper: https://arxiv.org/abs/2312.13313
* Code: https://github.com/woo525/ParamISP
**RecDiffusion: Rectangling for Image Stitching with Diffusion Models**
* Paper: https://arxiv.org/abs/2402.18192
* Code: https://github.com/lhaippp/RecDiffusion
**Residual Denoising Diffusion Models**
* Paper: https://arxiv.org/abs/2308.13712
* Code: https://github.com/nachifur/RDDM
**Real-Time Exposure Correction via Collaborative Transformations and Adaptive Sampling**
* Paper: https://arxiv.org/abs/2404.11884
* Code: https://github.com/HUST-IAL/CoTF
**SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image**
* Paper: https://arxiv.org/abs/2403.20018
* Code: https://github.com/WU-CVGL/SCINeRF
**Seeing Motion at Nighttime with an Event Camera**
* Paper: https://arxiv.org/abs/2404.11884
* Code: https://github.com/Liu-haoyue/NER-Net
**Shadow Generation for Composite Image Using Diffusion Model**
* Paper: https://arxiv.org/abs/2403.15234
* Code: https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBAv2
**Improving Spectral Snapshot Reconstruction with Spectral-Spatial Rectification**
* Paper: https://openaccess.thecvf.com/content/CVPR2024/html/Zhang_Improving_Spectral_Snapshot_Reconstruction_with_Spectral-Spatial_Rectification_CVPR_2024_paper.html
* Code: https://github.com/ZhangJC-2k/SSR
## 参考
相关Low-Level-Vision整理
Awesome-CVPR2020-Low-Level-Vision
Awesome-ECCV2020-Low-Level-Vision
Awesome-Low-Level-Vision-Research-Groups
Awesome-AIGC-Research-Groups

40
Docs/2024-11-15/report_1115.md Normal file
View File

@ -0,0 +1,40 @@
# 2024-11-15感知智能组会汇报
# 一、近期工作
**Image Generation** vs **Image Reconstruction**
* **Image Generation**：指通过算法从零开始生成新的图像，通常基于一些输入条件（如文本描述、特定样式、语义信息）或随机噪声（如生成对抗网络的输入）。目的是创造出逼真的、具有特定特征的图像。
* **无条件生成**是指无条件地从数据集中生成样本，即：$p(y)$；
* **条件图像生成**（子任务）是指根据标签有条件地从数据集中生成样本，即：$p(y|x)$
* **Image Reconstruction**：指从已有的、可能损坏、不完整或压缩的图像数据中恢复原始图像，或者从观测数据中重建图像。目标是尽可能还原出真实图像。
| **特性** | 图像生成 (Image Generation) | 图像重建 (Image Reconstruction) |
|--------|------|--------|
| **输入** | 随机噪声、文本或条件标签等 | 受损图像、不完整数据（如低分辨率图像、部分丢失的像素、模糊图像） |
| **输出** | 全新的图像，可能是艺术性的、合成的或逼真的 | 修复后的图像，接近原始清晰图像 |
| **目标** | 目标是创造性地生成新图像，关注生成图像的多样性和真实性；注重图像的视觉质量、逼真度以及与输入条件的匹配度。 | 目标是恢复或重建图像，尽可能减少噪声、模糊和失真；注重恢复的准确性和保真度，强调与真实图像的接近程度。 |
| **技术方法** | 典型方法包括生成对抗网络（GANs）、变分自编码器（VAEs）、扩散模型、条件生成模型等 | 通常基于优化和重建技术，包括卷积神经网络（CNNs）、自编码器（Autoencoders）、逆问题求解方法（如去噪、自适应插值）、正则化技术等 |
| **应用场景** | 数字艺术与内容创作（如DALL·E、Stable Diffusion）、数据增强（为训练AI生成多样化样本）、虚拟世界构建（如游戏、元宇宙）、图像翻译（风格迁移、照片转漫画） | 医学成像（如CT/MRI图像修复）、摄影中的图像去噪、超分辨率重建、遥感影像处理（如卫星图像云层去除）、逆向工程（如压缩图片的质量恢复） |
图像生成更侧重创新性，图像重建更注重还原性，但两者都在提升图像质量和智能处理方面发挥着重要作用。
我们所希望做的特征还原/图像补全/遮挡还原更倾向于**Image Reconstruction**部分的内容，但可以同时借鉴Generation和Reconstruction两个部分的内容展开。
* 从**Image Generation**的角度：关注补全的多样性和合理性
* 全局理解：
* MAE需要通过观察未遮挡部分来推测被遮挡区域的内容，这需要全局上下文信息的支持，与图像生成中的全局一致性思路相似。
* 补全结果不仅要像原图，还需要自然、符合上下文逻辑。
* 潜在表征学习：
* MAE利用自监督学习，类似生成任务中的生成网络，通过学习隐藏空间（latent space）中的表征来预测缺失区域的可能内容。
* 去噪补全：
* 与扩散模型类似，MAE可以从不确定性的潜在空间中逐步补全遮挡区域。
* 从**Image Reconstruction**的角度：注重还原真实性
* 输入部分：
* MAE的输入是遮挡了部分像素的图像，这与图像重建任务中不完整图像作为输入相似。
* 模型需要从局部上下文信息中恢复被遮挡的区域，这与图像重建强调从不完整数据中提取有用信息的思路一致。
* 目标：
* 恢复缺失的像素，使得补全后的图像尽可能接近原始图像，注重重建的保真度。
* MAE的设计目标是通过有效的编码和解码过程，使模型学会图像的局部与整体关系。
为了实现我们想要的部分，最好是找reconstruction领域去finetune的合理方法以及在generation领域的创新模块表示。除此之外，在segmentation领域找到优化的方案。
# 二、未来规划
阅读《Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting》以及所总结的和自己相关方向的论文

26
Docs/2024-11-22/Structure_Matters:_Tackling_the_Semantic_Discrepancy_in_Diffusion_Models_for_Image_Inpainting.md Normal file
View File

@ -0,0 +1,26 @@
# Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting
* Paper Link:https://openaccess.thecvf.com/content/CVPR2024/papers/Liu_Structure_Matters_Tackling_the_Semantic_Discrepancy_in_Diffusion_Models_for_CVPR_2024_paper.pdf
* Code Link:https://github.com/htyjers/StrDiffusion
## Abstract
图像修复中的去噪扩散概率模型（DDPMs）旨在正向修复过程中向图像纹理中添加噪声，反向修复过程中利用纹理未被掩盖区域恢复被掩盖区域去噪过程。尽管进行了有意义的语义生成，但现有的方法存在被屏蔽和未屏蔽区域之间的语义差异，因为语义密集的未屏蔽纹理在扩散过程中没有被完全退化，而被屏蔽区域在扩散过程中变成了纯噪声，导致它们之间存在较大的差异。本文旨在回答未掩码语义如何指导纹理去噪过程；以及如何解决语义差异，以促进一致和有意义的语义生成。为此，本文提出了一种基于结构引导的图像修复扩散模型StrDiffusion，该模型在结构指导下对传统的纹理去噪过程进行重新表述，从而得到一个简化的图像修复去噪目标，同时揭示了：
* 语义稀疏的结构有利于早期解决语义差异问题，而稠密的纹理则能在后期生成合理的语义；
* 得益于结构语义的时变稀疏性，未掩盖区域的语义本质上为纹理去噪过程提供了时变结构指导。在去噪过程中，通过利用掩码区域和未掩码区域之间的去噪结构的一致性，训练一个结构引导的神经网络来估计简化的去噪目标。
* 此外，设计了一种自适应重采样策略，作为结构是否能够指导纹理去噪过程的形式化标准，同时调节其语义相关性。通过大量实验验证了StrDiffusion算法相对于现有算法的优点。
## 1.Introduction
最近，图像修复支持了广泛的应用，例如照片恢复和图像编辑，其目的是用未掩蔽区域的语义信息恢复图像的掩蔽区域，其原理主要涵盖两个方面：掩蔽区域的合理语义及其与未掩蔽区域语义的一致性。现有的工作主要涉及基于扩散的[2,8]和基于补丁的[1,4,10]方案，该方案倾向于通过简单的颜色信息输入图像的小掩模或重复图案，而无法处理不规则或复杂的掩模。为了解决这个问题，大量的注意力[15,17,31,36]已经转移到卷积神经网络（CNN）上，它致力于在掩码区域周围对局部语义进行编码，而忽略来自未掩码区域的全局信息，导致远离掩码边界的区域保持毛茸。最近，自我注意机制[5，14，16，32，35]被提出以分割图像块的形式将掩蔽区域与未掩蔽区域全局关联，增强了它们之间的语义一致性。然而，这种策略在掩蔽区域内不同斑块之间的语义相关性较差。为此，利用语义稀疏结构[3，7，9，13，18，20，27，28，33，34]来增强它们的相关性，然而，这意味着严重依赖结构和纹理之间的语义一致性，因此不可避免地在内嵌输出中带有伪影。
幸运的是，去噪扩散概率模型（DDPM）[12,25]已成为强大的生成模型，在语义生成和模式收敛方面取得了显著进展，从而很好地弥补了图像修复语义生成不佳的问题[22,23,29,38]。具体来说，[22]建议采用预训练的DDPM作为先验，而不是专注于训练过程，并开发一种重采样策略来调节推理过程中的反向去噪过程。此外，[23]试图通过利用未掩蔽区域的语义来模拟图像修复的扩散过程，从而为去噪过程提供最佳的反向解决方案。这些方法大多表现出具有DDPM优势的语义有意义的修复结果，而忽略了掩蔽和未掩蔽区域之间的语义一致性。直觉是，语义密集的无掩模纹理退化为无掩模结构和高斯噪声的组合，而掩模区域在扩散过程中变成了纯噪声，导致它们之间存在很大差异（见图1（a）和图2（a）），并引发了以下问题：
* 1）*无掩模语义如何指导图像修复的纹理去噪过程？*我们的激励实验建议，当未掩蔽的语义与噪声相结合变得更稀疏时，例如，利用掩蔽图像的灰度或边缘图作为替代，差异问题在很大程度上得到了缓解，同时，在修复结果中损失了大量的语义信息；见图2（b）（c）。因此，未掩蔽区域随时间变化的不变语义无法很好地指导纹理去噪过程。
* 2）在 (1) 之后，人们自然会想知道*如何产生具有一致且有意义的语义的去噪结果*。很明显，稀疏结构在早期阶段有利于语义一致性，而密集纹理在去噪过程中倾向于在后期生成有意义的语义，这意味着去噪结果的一致性和合理语义之间的平衡。为了进一步产生理想的结果，我们将结构的引导视为纹理去噪过程的辅助；见图1(B)
## 2.Structure-Guided Texture Diffusion Models
### 2.1 Preliminaries: Denoising Diffusion Probabilistic Models for Image Inpainting
### 2.2 Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting
#### 2.2.1 Structure Matters: Sparse Semantics benefits the Semantic Consistency
#### 2.2.2 Optimal Solution to the Denoising Process Under the Guidance of the Structure
### 2.3 Structure-Guided Denoising Process: How does the Structure Guide the Texture Denoising Process?