Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC:...
Transcript of Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC:...
![Page 1: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/1.jpg)
Stephan Kopf Department of Computer Science IV
University of Mannheim, Germany
![Page 2: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/2.jpg)
Motivation
Part I: Basic Retargeting Operations
◦ Scaling and cropping
◦ Regions of interest
◦ Automatic crop & scale
◦ Sports video adaptation
Part II: Seam Carving
◦ Seam carving for images
◦ Preservation of straight lines
◦ Fast seam carving for videos
Summary
15.02.2011 Stephan Kopf 2
![Page 3: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/3.jpg)
Mobile phones are multimedia devices that allow to
◦ browse the Web
◦ display images and videos
◦ support novel input technologies (multi-touch)
15.02.2011 Stephan Kopf 3
But they still have limitations:
◦ Small screen size
◦ Wireless connection (bandwidth)
◦ Computational power (CPU, memory)
◦ Battery
![Page 4: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/4.jpg)
Typical resolutions of images and videos
◦ Digital camera: 10 megapixels (3.600 x 2.700 pixels)
◦ Camcorder: high definition (1.920 x 1.080 pixels)
◦ Mobile phone (240 x 320 pixels)
15.02.2011 Stephan Kopf 4
HD video mobile phone
Bitrate: 24 Mbit/s
Distortions caused by scaling (aspect ratio)
![Page 5: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/5.jpg)
Goals of media retargeting
Shrink photos and videos for the presentation on a mobile phone (this automatically limits the bitrate)
Keep aspect ratio
Preserve the most important visual content
Algorithms for image and video retargeting
15.02.2011 Stephan Kopf 5
![Page 6: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/6.jpg)
6
![Page 7: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/7.jpg)
Shrink image (merge pixels) by a fixed scale factor (uniform scaling)
Different scale factors for each axis change the aspect ratio (non-uniform scaling)
Relevance of image content is ignored
„Letterboxing“ is used to preserve aspect ratio
Example:
15.02.2011 Stephan Kopf 7
![Page 8: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/8.jpg)
Crop image borders until aspect ratios of image and display match
Relevance of image content is ignored: important content may be lost
Typically use scaling to convert to target size
Example:
15.02.2011 Stephan Kopf 8
![Page 9: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/9.jpg)
Idea
Identify most relevant image regions (regions of interest)
Crop borders but preserve regions of interest
Use automatic algorithms to identify regions of interest:
◦ Saliency maps
◦ Faces
◦ Text regions
15.02.2011 Stephan Kopf 9
![Page 10: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/10.jpg)
Assumption: image regions that are relevant for an observer have a high contrast
Step 1: Contrast map of an image of size n × m :
color of a pixel: pi ,j
pixel in local neighborhood of pi ,j :
distance function: d (.)
Step 2: Quantize contrast map
Step 3: Find connected regions
Step 4: Mark region of interest
15.02.2011 Stephan Kopf 10
*Source: Ma and Zhang HJ: Contrast-based image attention analysis by using fuzzy growing, ACM Intl. Conf. on Multimedia, 2003
![Page 11: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/11.jpg)
contrast map quantized contrast map
region of interest bounding box
15.02.2011 Stephan Kopf 11
![Page 12: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/12.jpg)
Use automatic face detection algorithms to localize face regions
Frontal face detection algorithms work very robust (in contrast to face recognition)
15.02.2011 Stephan Kopf 12
![Page 13: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/13.jpg)
Characteristic features of text:
◦ horizontal alignment
◦ significant luminance difference between text and background
◦ the character size is within a certain range
◦ single-colored
◦ text is visible in consecutive frames (video)
◦ horizontal or vertical motion is possible (video)
Calculate a horizontal projection profile to detect the boundaries of text lines
15.02.2011 Stephan Kopf 13
![Page 14: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/14.jpg)
Calculate importance value V for each region of size H:
minimum perceptible size: Hmin
maximum reasonable size: Hmax
Find optimal target region W based on regions of interest Si:
15.02.2011 Stephan Kopf 14
![Page 15: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/15.jpg)
Selection of one feature
Combination of two features
… three features Full image
15.02.2011 Stephan Kopf 15
![Page 16: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/16.jpg)
scaling
cropping
crop & scale
15.02.2011 Stephan Kopf 16
![Page 17: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/17.jpg)
Automatically detect:
Court lines
Players
Ball
scaled video
modify video content
15.02.2011 Stephan Kopf 17
*Source: Kopf, Guthier, Farin, Han: Analysis and Retargeting of Ball Sports Video, IEEE Workshop on Applications of Computer Vision, 2011
![Page 18: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/18.jpg)
Step 1: Mark bright pixels (line pixels)
Step 2: Algorithm to detect straight lines (based on RANSAC)
1. Randomly select two line pixels and calculate line parameters
2. Count number of white pixels N located on line
3. If (N > threshold) stop
4. Goto 1.
Step 3: Remove line pixels and detect next line (Step 2)
15.02.2011 Stephan Kopf 18
RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications ACM, vol 24(6), 1981.
![Page 19: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/19.jpg)
Problem: Position of lines change from frame to frame
Solution: use a reference court model to estimate camera motion
◦ Step 1: Calculate intersection points of two lines
◦ Step 2: Transform lines to court model
How many intersection points do we need for the transformation?
15.02.2011 Stephan Kopf 19
![Page 20: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/20.jpg)
Translation (horizontal/vertical shift)
1 intersection point
15.02.2011 Stephan Kopf 20
Translation and scaling
2 intersection points
Affine transform (translation, scaling, rotation)
3 intersection points
Perspective transform
4 intersection points
![Page 21: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/21.jpg)
cropping scaling crop & scale
(zoom on largest player) modify lines & ball
15.02.2011 Stephan Kopf 21
![Page 22: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/22.jpg)
22
![Page 23: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/23.jpg)
If important content is located near image borders: crop & scale is not applicable
Idea of seam carving*
Systematic removal of less important pixels
Use energy function as measurement of „importance“ of single pixels
*Source: Shai Avidan and Ariel Shamir: Seam Carving for Content-Aware Image Resizing. ACM SIGGRAPH, 2007
15.02.2011 Stephan Kopf 23
![Page 24: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/24.jpg)
Image width should be reduced by 40 percent
original image energy map
15.02.2011 Stephan Kopf 24
![Page 25: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/25.jpg)
Remove N pixels with the lowest energy from each line
15.02.2011 Stephan Kopf 25
remove N=200 pixels from each line
based on energy values
source image
![Page 26: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/26.jpg)
Summarize energy in each column of the image and remove N columns with lowest energy
remove 200 columns based on energy values
of columns
original image
15.02.2011 Stephan Kopf 26
![Page 27: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/27.jpg)
A vertical seam is an 8-connected path of pixels from top to bottom that contains one and only one pixel in each row.
Formal definition:
Horizontal seams are defined in a analog way.
15.02.2011 Stephan Kopf 27
1|1)-x(i-x(i)| :i subject to ,i)}{(x(i), =}{s=s n
1i
n
1i
x
i
x
![Page 28: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/28.jpg)
Advantage of seams compared to columns or rows:
◦ Pixels of low energy are removed
◦ Relevant objects are preserved
15.02.2011 Stephan Kopf 28
![Page 29: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/29.jpg)
Remove the vertical seam with the lowest energy
Repeat this step N times
15.02.2011 Stephan Kopf 29
remove N=200 seams based on lowest energy
source image
![Page 30: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/30.jpg)
Seam carving uses an energy function that characterizes the relevance of each pixel (similar to saliency maps).
The optimal seam minimizes the cumulated pixel energy of all seam pixels.
Method to find optimal seam: dynamic programming
15.02.2011 Stephan Kopf 30
![Page 31: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/31.jpg)
M ( i, j ) specifies the cost of the optimal (vertical) seam from the upper image border to pixel position (i, j )
Calculate M( i, j ) recursively:
)1,1(
),1(
)1,1(
min),(),(
jiM
jiM
jiM
jiejiM
15.02.2011 Stephan Kopf 31
![Page 32: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/32.jpg)
1
Example how to calculate the optimal seam:
1
3
6
7
3
6
7
2 5
1
4
1
2
3
4
1
2
3
3
5
4
4
1
)1,1(
),1(
)1,1(
min),(),(
jiM
jiM
jiM
jiejiM
2 5
4
3
4
5
4
5
7
9
8
9
15.02.2011 Stephan Kopf 32
energy map cumulated energy map M( i, j )
![Page 33: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/33.jpg)
Image gradient: simple energy function that calculates the luminance difference to adjacent pixels:
Assumption: Luminance values do not differ much in image regions of low relevance
This simple energy function gives good results in many cases
),(),()),(( yxIy
yxIx
yxIe
15.02.2011 Stephan Kopf 33
![Page 34: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/34.jpg)
Problem: The light house is an important region, but the pixel values are very similar
original image optimal seams result
15.02.2011 Stephan Kopf 34
![Page 35: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/35.jpg)
Combine energy function with saliency map
)),((),()),(( yxIeyxsaliencywyxIe ssal
saliency map optimal seams result (esal is used as energy function)
Source: Hwang and Chien. Content-Aware Image Resizing using Perceptual Seam Carving with Human Attention Model. IEEE Conference on Multimedia and Expo, 2008.
15.02.2011 Stephan Kopf 35
![Page 36: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/36.jpg)
Use results from face detection as additional saliency:
)),((),(),()),(( yxIeyxfacewyxsaliencywyxIe fsfacesal
saliency map
face map
seams based on esal+face as
energy function
result
15.02.2011 Stephan Kopf 36
![Page 37: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/37.jpg)
The quality of seam carving drops significantly in case of straight lines
original image seam carving (width reduced by 40%)
Source: Kiess, Kopf, Guthier and Effelsberg: Seam Carving with Improved Edge Preservation. Proc. of IS&T/SPIE Electronic Imaging, 2010.
15.02.2011 Stephan Kopf 37
![Page 38: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/38.jpg)
Problem: lines become distorted when seams are removed
image section visualizing a straight line
seams intersect a straight line
result after removal of seams
15.02.2011 Stephan Kopf 38
![Page 39: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/39.jpg)
This is especially critical when several seams intersect a line in adjacent pixel positions
Idea: Distribute intersection points of seams and lines along the line
seams intersect a line in adjacent pixel positions
result: errors are clearly visible
equal distribution of intersection
points
result: errors are much less obvious
15.02.2011 Stephan Kopf 39
![Page 40: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/40.jpg)
Implementation: modify energy function before the next optimal seam is calculated
Intersection point of seam and line: increase energy values in a certain radius
The following seams will avoid these pixels
seam intersects a straight line
Modify energy function next to the intersection
point
detect next seams and modify energy function for each
intersection
15.02.2011 Stephan Kopf 40
![Page 41: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/41.jpg)
original image seam carving seam Carving with line preservation
15.02.2011 Stephan Kopf 41
![Page 42: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/42.jpg)
1. Idea: Use seam carving on each frame separately video becomes blurred and shaky
original adapted
![Page 43: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/43.jpg)
Improvements
Video defines a 3D space-time volume (3D cube)
Remove 2D seam manifolds (seam surface areas) where each seam pixel is connected in 3D
Use graph cuts (max-flow min-cut) to detect optimal seam manifold
Source: Rubinstein, Shamir, Avidan: Improved seam carving for video retargeting. ACM Trans. Graph. 27, 3, 2008.
time
source node
sink node
frame N
frame 1 edges: energy between pixels
Computational effort?
![Page 44: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/44.jpg)
Idea
Create one image that aggregates the pixel values / energy values of all frames
Detect 1D seam in aggregated image
Map this seam back to all frames
Source: Kopf, Kiess, Lemelson, Effelsberg: FSCAV -
Fast Seam Carving for Size Adaptation of Videos,
ACM Intl. Conf. on Multimedia, 2009.
15.02.2011 Stephan Kopf 44
![Page 45: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/45.jpg)
Problem: camera motion, zoom, panning
Use image registration techniques to calculate the parameters of the camera model (use perspective camera model)
Align frames and create a background image
Detect optimal seam in background image
Use inverse camera motion to transform optimal seam back to all original frames
15.02.2011 Stephan Kopf 45
![Page 46: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/46.jpg)
Example: construct background image from a camera pan
15.02.2011 Stephan Kopf 46
![Page 47: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/47.jpg)
15.02.2011 Stephan Kopf 47
scaling seam carving
![Page 48: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/48.jpg)
15.02.2011 Stephan Kopf 48
scaling fast seam carving
![Page 49: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/49.jpg)
The quality of adapted images or videos depends on the visual content. The results of crop & scale might be much better than seam carving or vice versa.
Crop & scale typically works well if the relevant content is located in a small region.
In case of large background areas, many seams with low energy are detected and the results based on seam carving are very good.
15.02.2011 Stephan Kopf 49
No technique works well if most of the content is highly relevant.
![Page 50: Stephan Kopf Department of Computer Science IV University ......Stephan Kopf 15.02.2011 18 RANSAC: Fischler, Bolles: Random sample concensus: a paradigm for model fitting with applications](https://reader036.fdocuments.net/reader036/viewer/2022081601/611dcc528530fa1bef0ef74e/html5/thumbnails/50.jpg)
Would it be possible to find better energy functions for seams?
Would it be possible to preserve other geometric objects similar to straight lines?
Would it be possible to automatically evaluate the quality of adapted images or videos?
15.02.2011 Stephan Kopf 50