MACHINE VISION GROUP Graphics hardware accelerated panorama builder for mobile phones Miguel...

MACHINE VISION GROUP

Graphics hardware accelerated panorama builder for mobile phones

Miguel Bordallo López*, Jari Hannuksela*, Olli Silvén* and Markku Vehviläinen**,

* University of Oulu, Finland** Nokia Research Center, Tampere, Finland


Contents

IntroductionDocument Panorama Builder

• Interactive user interface• Quality evaluation and frame selection• Re-registration and blending• Computation times

Scene Panorama BuilderGraphics processor as a computing engineGPU accelerated panorama builder


IntroductionDocument capture with single snapshot

ViewFinder Image Real size section


IntroductionPanorama Builder

Panorama builder• Used to combine several images into one wide-angle mosaic image• Traditionally image mosaic is formed from a few separate images• Video sequences can also be used as a source


IntroductionVideo Mosaicking

• Videos contain lots of information – most of it redundant• Video mosaicking vs. image mosaicking

– image registration is easier due to smaller motions between frames

– some frames can be discarded without losing information


Document Panorama BuilderInteractive user interface

•VGA video analysis

•Motion estimation system calculates shift, rotation and scale in real time

•When frame is suitable for stitching (high quality), the user receives feedback and instructions


Document Panorama BuilderQuality determination and frame selection


Document Panorama BuilderRe-registration and blending

Selected frames

Final panorama


Document Panorama BuilderComputation times

• 1. Registration with SIFT feature based RANSAC to find the warping and stitching parameters– approx. 6 frames/minute in software

• 2. Image Correction: warping with interpolation – approx. 2 frames/s in software

• 3. Quality determination and frame selection during the capture stage – approx. 10 frames/s in software

• 4. Motion estimation during capture stage – approx. 30 frames/s in software


Scene Panorama Builder

• >360 degrees panoramas from landscape• Real-time registration• Real-time frame evaluation and selection• Real-time frame correction• On-the-fly stitching• Less memory requirements


GPU as a computing engine

• Newer phones include a GPU chipset

• OpenGL ES as a highly optimized and attractive accelerator interface

• Compatible data formats for graphics and camera sub-systems desirable

• Emerging platforms (CUDA , OpenCL) facilitate using the GPU as a computing resource


Fixed pipeline (OpenGL ES 1.1) vs. programmable pipeline (OpenGL ES 2.0)


Stream processing (OpenGL) vs. shared memory processing (CUDA)


OpenCL (Embedded Profile)

• Emerging platforms will offer needed flexibility

• Code executed concurrently on CPU & GPU (even DSP)

• High parallelization on image processing computations . High efficiency


GPU-accelerated panorama builder

• Open GL ES 1.1:– Warping with interpolations:

• >4 times faster

– Pixel blending:

• Open GL ES 2.0:– Image features (SIFT, Haar,...) extraction:

• 10 times faster on PC, CPU load decreased 50%

– Frame quality evaluation

• Emerging platforms:– RANSAC estimator– Better efficiency– Enabling the use of other chipsets


Image warping with interpolation on OpenGL ES 1.1

Create quad

Transform

Render texture


Summary

•Document mosaicing allows feedback to the user•Quality can be measured while capturing•Mobile phones are an excellent platform for this method

•GPUs can be used as a general purpose procesors•New platforms will offer more efficiency and flexibility

•Not optimized interfaces include excesive overheads•Lack of RAM in small devices sitll a problem

MACHINE VISION GROUP Graphics hardware accelerated panorama builder for mobile phones Miguel...

Documents

Transcript of MACHINE VISION GROUP Graphics hardware accelerated panorama builder for mobile phones Miguel...