Recovering Realistic Texture in Image Super-resolution by...
Transcript of Recovering Realistic Texture in Image Super-resolution by...
![Page 1: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/1.jpg)
Recovering Realistic Texture in Image Super-resolution by
Deep Spatial Feature Transform
Xintao Wang Ke Yu Chao Dong Chen Change Loy
![Page 2: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/2.jpg)
Problem
Low-resolution image High-resolution image
enlarge 4 times
![Page 3: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/3.jpg)
Previous work• Contemporary SR algorithms are mostly CNN-based methods[1].
• Most of CNN-based methods use pixel-wise loss function. (MSE-based model) good at recovering edges and smooth areas
not good at texture recovery
• Adversarial loss is introduced in SRGAN[2] and EnhanceNet[3]. (GAN-based model) encourage the network to favor solutions that look more like natural images
visual quality of reconstruction is significantly improved
SRCNN SRGAN Ground-truth
[1] C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In ECCV, 2014.[2] C. Ledig, L. Theis, F. Husz ar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, 2017.[3] M. S. Sajjadi, B. Sch olkopf, and M. Hirsch. EnhanceNet: Single image super-resolution through automated texture synthesis. In ICCV, 2017.
![Page 4: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/4.jpg)
Motivation
building x4
plant x4swap priors
plant prior
building prior
![Page 5: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/5.jpg)
animal
building water
sky
grass
mountain
plant
Semantic categorical prior
![Page 6: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/6.jpg)
Issues
1. How to represent the semantic categorical prior?
2. How categorical prior can be incorporated into the reconstruction process effectively?
Our approach: explore semantic segmentation probability maps as the categorical prior up to pixel level.
Our approach: propose a novel Spatial Feature Transform that is capable of altering the network behavior conditioned on other information.
![Page 7: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/7.jpg)
Represent categorical prior• Contemporary CNN segmentation network[1]
• fine-tuned on LR images
ResNet 101
𝐾 categories
𝑎𝑟𝑔𝑚𝑎𝑥
semantic categorical prior
probability maps
[1] Z. Liu, X. Li, P. Luo, C.-C. Loy, and X. Tang. Semantic image segmentation via deep parsing network. In ICCV, 2015.
![Page 8: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/8.jpg)
Input LR images
animal
sky
grass
building
mountain
plant
water
background
Segments on LR images
Segments on HR images
Ground-truth
Examples on segmentation
![Page 9: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/9.jpg)
𝒚 = 𝐺𝜃(𝒙)
Ψ = (𝑃1, 𝑃2, … , 𝑃𝐾)
Incorporate conditions
CNN for SR
𝒚 = 𝐺𝜃(𝒙)
input LR image
𝒙restored image
𝒚𝑛𝑒𝑡 𝐺
𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑧𝑒𝑑 𝑏𝑦 𝜃
Categorical prior
Ψ = (𝑃1, 𝑃2, … , 𝑃𝐾)
𝒚 = 𝐺𝜃(𝒙|Ψ)probability maps
𝑃1, 𝑃2, … , 𝑃𝐾 ?prior
Ψ
![Page 10: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/10.jpg)
Spatial Feature Transform
• By learning a mapping function ℳ, the prior Ψ is modeled by a pair of affine transformation parameters (𝛾, 𝛽) .
• The modulation is then carried out by an affine transformation on feature maps 𝑭.
ℳ:Ψ ↦ (𝛾, 𝛽)
SF𝑇 𝑭 𝛾, 𝛽 = 𝛾⨀𝑭+ 𝛽
𝒚 = 𝐺𝜃(𝒙|Ψ)ℳ:Ψ ↦ (𝛾, 𝛽)
SFT 𝑭 𝛾, 𝛽 = 𝛾⨀𝑭+ 𝛽
𝒚 = 𝐺𝜃(𝒙|𝛾, 𝛽)
![Page 11: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/11.jpg)
Spatial Feature Transform
Co
nv
Res
idu
al b
lock
SFT
laye
r
SFT
laye
r
Co
nv
Co
nv
Residual block
Res
idu
al b
lock
SFT
laye
r
Co
nv
Up
sam
plin
g
Co
nv
Co
nv
Co
nv
⨀ +C
on
v
Co
nv
Co
nv
Co
nv
SFT layer
conditions
features 𝜸𝑖 𝜷𝑖Co
nv
Co
nv
Co
nv
Co
nv
Condition Network
Segmentationprobability
maps
Shared SFT conditions
![Page 12: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/12.jpg)
loss function
• Adversarial loss[1]
min𝜃
max𝜂
Ε𝑦~𝑝HR 𝑙𝑜𝑔𝐷𝜂 𝑦 + Ε𝑥~𝑝LRlog(1 − 𝐷𝜂 𝐺𝜃(𝑥) )
Generator
Discriminator
Compete
• Perceptual loss[2]
encourage the network to generate images that reside on the manifold of natural images
𝜙𝑉𝐺𝐺 𝑦 − 𝜙𝑉𝐺𝐺 𝑦 22
use a pre-trained 19-layer VGG network (features before conv54)
optimize a super-resolution model in a feature space
[1] Goodfellow, Ian, et al. Generative adversarial nets. In NIPS. 2014.[2] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
![Page 13: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/13.jpg)
𝑷𝒃𝒖𝒊𝒍𝒅𝒊𝒏𝒈 map
RestoredLR patch
𝑷𝒈𝒓𝒂𝒔𝒔 map 𝜸 map of 𝐶6 𝜷 map of 𝐶7Input
Spatial condition
• The modulation parameters (𝛾, 𝛽) have a close relationship with probability maps 𝑷 and contain spatial information.
![Page 14: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/14.jpg)
𝜸 map of 𝐶51
Restored
LR patch 𝑷𝒑𝒍𝒂𝒏𝒕 map
𝑷𝒈𝒓𝒂𝒔𝒔 map
𝜷 map of 𝐶1
𝜸 map of 𝐶14 𝜷 map of 𝐶5
Delicate modulation
![Page 15: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/15.jpg)
Results
GTSRCNN SRGAN EnhanceNet SFT-Net (ours)
PSNR: 24.83dB PSNR: 23.36dB PSNR: 22.71dB PSNR: 22.90dB
![Page 16: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/16.jpg)
Bicubic SRCNN VDSR LapSRN DRRN MemNet EnhanceNet SRGAN SFT-Net (ours) GT
Results
MSE-based method GAN-based method
![Page 17: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/17.jpg)
54.5
76.468 75
56.468.7 65.7
sky building grass animal plant water mountain
67 33
85 15Ours
Ours
EnhanceNet
SRGAN
User study – part I
![Page 18: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/18.jpg)
User study – part II
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Rank-1 Rank-2 Rank-3 Rank-4
80.4 18.4
18.6 79.6
61.3 36.3
37 62.4
GT
Ours
MemNet
SRCNN
![Page 19: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/19.jpg)
buildingprior
skyprior
grassprior
mountainprior
waterprior
plantprior
animalpriorbicubic
bu
ildin
gsk
ygr
ass
mo
un
tain
wat
er
pla
nt
anim
al
Impact of different priors
building sky grass mountain water plant animalbuilding sky grass mountain water plant animal
building prior
![Page 20: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/20.jpg)
buildingprior
skyprior
grassprior
mountainprior
waterprior
plantprior
animalpriorbicubic
bu
ildin
gsk
ygr
ass
mo
un
tain
wat
er
pla
nt
anim
al
Impact of different priors
buildingprior
skyprior
grassprior
mountainprior
waterprior
plantprior
animalpriorbicubicmountain
![Page 21: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/21.jpg)
Other conditioning methods
[1] S. Zhu, S. Fidler, R. Urtasun, D. Lin, and C. C. Loy. Be your own prada: Fashion synthesis with structural coherence. In ICCV, 2017.[2] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. FiLM: Visual reasoning with a general conditioning layer. arXiv preprint arXiv:1709.07871, 2017.
Compositionalmapping[1]
FiLM[2]Input concatenation
![Page 22: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/22.jpg)
Input concatenation
Compositionalmapping
FiLMSFT-Net (ours)
Comparison with other conditioning methods
![Page 23: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/23.jpg)
SRGAN SRGANOurs Ours
Robustness to out-of-category
![Page 24: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/24.jpg)
Conclusion
• Explore semantic segmentation maps as categorical prior for realistic texture recovery.
• Propose a novel Spatial Feature Transform layer to efficiently incorporate the categorical conditions into a CNN-based SR network.
• Extensive comparisons and a user study demonstrate the capability of SFT-Net in generating realistic and visually pleasing textures.
![Page 25: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/25.jpg)
Crafting a Toolchain for Image Restoration by
Deep Reinforcement Learning
Ke Yu Chao Dong Liang Lin Chen Change Loy
![Page 26: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/26.jpg)
Image Restoration
• There are many individual tasks• Denoising• Deblurring• JPEG Deblocking• Super-Resolution• …
• Towards more complicated distortions• Address multiple levels of degradation in one task[1, 2]
• Address multiple individual tasks[3]
![Page 27: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/27.jpg)
Image Restoration – A New Setting
• Consider multiple distortions simultaneously • Real-world: Image capture and storage• Synthetic: Gaussian blur, Gaussian noise and JPEG compression
GaussianBlur
GaussianNoise
JPEGCompression
Real-worldScenario
SyntheticSetting
Our New Task
![Page 28: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/28.jpg)
Motivation
• Can we use a single CNN to address multiple distortions?• Inefficient: Require a huge network to handle all the possibilities• Inflexible: All kinds of distorted images are processed with the
same structure
• Find a more efficient and flexible approach!• Process different distortion in a different way
![Page 29: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/29.jpg)
Method – Decision Making
• Progressively restore the image quality
• Treat image restoration as a decision making process
Noisy! Try a denoising toolBlurry! Try a
deblurring toolArtifacts! Try a
deblocking toolGood enough :)
![Page 30: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/30.jpg)
Method – Overview
• Our framework requires a toolbox and an agent
Agent Agent
Toolbox Toolbox
![Page 31: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/31.jpg)
Method – Toolbox
• We design 12 tools, each of which addresses a simple task• 3-layer CNN[4]
• 8-layer CNN
![Page 32: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/32.jpg)
Method – Agent
• Use reinforcement learning to address tool selection
Statecurrent distorted image
action at last step
12 tools
stoppingAction
Reward: PSNR gain at each step
InputImage
FeatureExtractor
One-hot Encoder
LSTM 𝒗1
𝐈1
𝑆1Agent
𝒗1
Structure:
![Page 33: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/33.jpg)
Method – Joint Training
• Challenge of ‘Middle State’• Intermediate results after several steps of processing• None of the tools has seen these intermediate results
• Joint Training
...
...
...
forward
backwardtoolchain 1
toolchain 2forward
backward
MSE loss
MSE loss
![Page 34: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/34.jpg)
Experimental Results
• Dataset: DIV2K[5]
• Comparison with generic models for image restoration• VDSR[1]
• DnCNN[3]
![Page 35: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/35.jpg)
Experimental Results
• Quantitative results on DIV2K
• Runtime Analyses
More efficient
Better generality
Competitive performance
![Page 36: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/36.jpg)
Experimental Results
• Qualitative results on DIV2K
Input
1st step
2nd step
3rd step
VDSR-s
VDSR[1]
Mild (unseen) Moderate Severe (unseen)
![Page 37: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/37.jpg)
Experimental Results
• Qualitative results on real-world imagesInput 1st step 2nd step 3rd step VDSR[1]
![Page 38: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/38.jpg)
Experimental Results
• Ablation StudyJoint training
Stopping action
![Page 39: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/39.jpg)
Conclusion
• Contributions• Address image restoration in a reinforcement learning framework• Propose joint learning to cope with middle processing state• Dynamically formed toolchain performs competitively against
human-designed networks with less computational complexity
• Future work• Incorporate more tools (trained with GAN loss)• Handle spatial-variant distortions
![Page 40: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/40.jpg)
Thanks!Q & A
![Page 41: Recovering Realistic Texture in Image Super-resolution by ...ice.dlut.edu.cn/valse2018/ppt/2018ValseDChao.pdf · Recovering Realistic Texture in Image Super-resolution by ... Residual](https://reader030.fdocuments.net/reader030/viewer/2022041009/5eb675a1e39ca4630f35be2c/html5/thumbnails/41.jpg)
Reference
[1] J. Kim, J. Kwon Lee, and K. Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.
[2] Y. Tai, J. Yang, X. Liu, and C. Xu. Memnet: A persistent memory network for image restoration. In ICCV, 2017.
[3] K. Zhang,W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. TIP, 2017.
[4] C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. TPAMI, 38(2):295–307, 2016.
[5] E. Agustsson and R. Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPR Workshop, 2017.