Whiteboard Scanning Using Super-

resolutionWode Ni

Advisor: John MacCormickCOMP 491 Final Presentation

Dec 18 2015

OUTLINE OF THE PRESENTATION• Motivation• Background – image formulation• Superresolution algorithm• OpenCV• Experiment results• Summary of this semester’s work• Future work

MOTIVATION• The “distance whiteboard” scenario: far away from the whiteboard, try to obtain a clear, scanned image of the whiteboard• Difficulty: limitation of camera

resolution• Proposed method: Take a video,

or multiple images and apply superresolution algorithm on the set of frames of the video/images and get an image of high resolution.

LR image scanned by CamScanner

BACKGROUND

Mitzel, Dennis, et al. "Video super resolution using duality based tv-l 1 optical flow." Pattern Recognition. Springer Berlin Heidelberg, 2009. 432-441.Symbol Degrading

FactorsReasons

D Downsampling The sensor of digital camera (typically CMOS)B Blurring Optical lens of the camera or atmospheric

effectW Warping Motion of the cameraman e Additive noise Usually introduced by high ISO

BACKGROUND - continued

• Goal: Given k Low-Resolution(LR) images(), try to calculate a larger image () with high-resolution(HR) and desirably higher quality.• Method: “fuse” k LR images with respect to some “energy

function”. In the context of SR algorithms, this function will return a higher value if the estimation of the HR image is “different” from the LR images. Thus, we are trying to minimize the energy function.• For the brevity of this presentation, the design of the energy

function and derivation of the minimization is omitted.

SUPERRESOLUTION ALGORITHM• Motion estimation:

optical flow• Need to enlarge

the motion matrix using interpolation (linear)• The algorithm will

run through T times, and compute an estimation of HR image at a time.• At the end of each

iteration, the result is updated.

OPENCV

• Open source Computer Vision Library: now maintained by Itseez• As for 3.0.0 version: 29 main modules, which includes superres.

Extra modules contributed are also available• Built-in data structures, memory management, and hundreds of

computer vision algorithms available.• Superres module uses methods from both Farsiu 2004 and Mitzel

2009.

SUPERRES Name in OpenCV Symbol

in Mitzel09

Meaning

Temporal Radius Area N The number of LR images to be processed. Iterations T The number of HR estimations to be computed.• Temporal Radius Area: does not exist in Mitzel’s

paper. The OpenCV version of SR algorithm can output a video!

• As shown in Status Report I presentation, the effect on running time when increasing these parameters linear, meaning the same amount of work is done on each image and in each iteration.

• Result of running the algorithm: Demo

Running time of SR algorithm with varying parameters

SOME NEW RESULTS

• Computed Sum of Squared Difference(SSD) on SR result on the “synthesized image”.• Increasing

iterations: First decrease and increase• Increasing temporal

area: increase and converge• Other error metrics?

WHAT I DID THIS SEMESTER…

Tried to estimate the quality of the output quantitatively

Studied the source code and understood more about the details of the implementation

Did some black-box analysis on the OpenCV implementation of SR algorithm: running times of different values of the parameters

Found existing SR implementations and decided to work with OpenCV

Summer: studied basics of Computer Vision, C++, Linear Algebra, and OpenCV.

FUTURE WORK• Coming back to what motivates the project: if we

put the result of SR algorithm in CamScanner, will the quality of the scanned image be improved? How can we make the result even better? What is the computational cost and quality improvement of applying SR?• Deeper into OpenCV: do some serious debugging

(possibly modifying the source) and investigate the reason behind the weird SSD curve.• Additional work: Study more math and try to

understand the theoretical basis of SR better.• Other comments and thoughts

LR scanned image

SR scanned image

Thank You!

• References:• Dennis Mitzel, Thomas Pock, Thomas Schoenemann, and Daniel

Cremers. Video super resolution using duality based tv-l 1 optical flow. In Pattern Recognition, pages 432–441. Springer, 2009.

• Sina Farsiu, M Dirk Robinson, Michael Elad, and Peyman Milanfar. Fast and robust multiframe super resolution. Image processing, IEEE Transactions on, 13(10):1327–1344, 2004.