Multi-view video based multiple objects segmentation using graph cut and spatiotemporal projections...
-
Upload
shanon-owens -
Category
Documents
-
view
219 -
download
2
Transcript of Multi-view video based multiple objects segmentation using graph cut and spatiotemporal projections...
Multi-view video based multiple objects segmentation using graph cut and spatiotemporal projections
Journal of Visual Communication and Image Representation
Volume 21 Issues 5ndash6 JulyndashAugust 2010 Pages 453ndash461
Qian Zhang King Ngi Ngan
Department of Electronic Engineering The Chinese University of Hong Kong Shatin New Territories Hong Kong
Speaker Yi-Ting Chen
2 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
3 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
4 Introduction
Most of the interest has been focused on the research of single view segmentation
Depth information in the 3D scene can be reconstructed from multi-view images but multiple view segmentation has not attracted much attention
Most of the classical and start-of-the-art graph cut based segmentation algorithms require userrsquos interventions to specify the initial foreground and background regions as hard constraints
5 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
6 Overview of the proposed framework We built a five-view camera system for views v 01234isin
To reduce the projection error and avoid extensive computational load we select view 2 as the key view to start the segmentation process
7 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
8 Automatic initial interested objects (IIOs) extraction based on saliency model (SM)
Inspired by the work in [33] more sophisticated cues such as motion and depth are combined into our topographical SM
(a) input image (b) saliency map using depth and motion and (c) extracted IIOs
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
9 Multiple objects segmentation using graph cut
For individual object we construct a sub-graph for the pixels belonging to its ldquoObject Rectanglerdquo an enlarged rectangle to encompass the whole object
and restricts the segmentation region
we convert multiple objects segmentation into several sub-segmentation problems
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
2 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
3 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
4 Introduction
Most of the interest has been focused on the research of single view segmentation
Depth information in the 3D scene can be reconstructed from multi-view images but multiple view segmentation has not attracted much attention
Most of the classical and start-of-the-art graph cut based segmentation algorithms require userrsquos interventions to specify the initial foreground and background regions as hard constraints
5 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
6 Overview of the proposed framework We built a five-view camera system for views v 01234isin
To reduce the projection error and avoid extensive computational load we select view 2 as the key view to start the segmentation process
7 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
8 Automatic initial interested objects (IIOs) extraction based on saliency model (SM)
Inspired by the work in [33] more sophisticated cues such as motion and depth are combined into our topographical SM
(a) input image (b) saliency map using depth and motion and (c) extracted IIOs
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
9 Multiple objects segmentation using graph cut
For individual object we construct a sub-graph for the pixels belonging to its ldquoObject Rectanglerdquo an enlarged rectangle to encompass the whole object
and restricts the segmentation region
we convert multiple objects segmentation into several sub-segmentation problems
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
3 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
4 Introduction
Most of the interest has been focused on the research of single view segmentation
Depth information in the 3D scene can be reconstructed from multi-view images but multiple view segmentation has not attracted much attention
Most of the classical and start-of-the-art graph cut based segmentation algorithms require userrsquos interventions to specify the initial foreground and background regions as hard constraints
5 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
6 Overview of the proposed framework We built a five-view camera system for views v 01234isin
To reduce the projection error and avoid extensive computational load we select view 2 as the key view to start the segmentation process
7 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
8 Automatic initial interested objects (IIOs) extraction based on saliency model (SM)
Inspired by the work in [33] more sophisticated cues such as motion and depth are combined into our topographical SM
(a) input image (b) saliency map using depth and motion and (c) extracted IIOs
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
9 Multiple objects segmentation using graph cut
For individual object we construct a sub-graph for the pixels belonging to its ldquoObject Rectanglerdquo an enlarged rectangle to encompass the whole object
and restricts the segmentation region
we convert multiple objects segmentation into several sub-segmentation problems
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
4 Introduction
Most of the interest has been focused on the research of single view segmentation
Depth information in the 3D scene can be reconstructed from multi-view images but multiple view segmentation has not attracted much attention
Most of the classical and start-of-the-art graph cut based segmentation algorithms require userrsquos interventions to specify the initial foreground and background regions as hard constraints
5 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
6 Overview of the proposed framework We built a five-view camera system for views v 01234isin
To reduce the projection error and avoid extensive computational load we select view 2 as the key view to start the segmentation process
7 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
8 Automatic initial interested objects (IIOs) extraction based on saliency model (SM)
Inspired by the work in [33] more sophisticated cues such as motion and depth are combined into our topographical SM
(a) input image (b) saliency map using depth and motion and (c) extracted IIOs
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
9 Multiple objects segmentation using graph cut
For individual object we construct a sub-graph for the pixels belonging to its ldquoObject Rectanglerdquo an enlarged rectangle to encompass the whole object
and restricts the segmentation region
we convert multiple objects segmentation into several sub-segmentation problems
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
5 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
6 Overview of the proposed framework We built a five-view camera system for views v 01234isin
To reduce the projection error and avoid extensive computational load we select view 2 as the key view to start the segmentation process
7 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
8 Automatic initial interested objects (IIOs) extraction based on saliency model (SM)
Inspired by the work in [33] more sophisticated cues such as motion and depth are combined into our topographical SM
(a) input image (b) saliency map using depth and motion and (c) extracted IIOs
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
9 Multiple objects segmentation using graph cut
For individual object we construct a sub-graph for the pixels belonging to its ldquoObject Rectanglerdquo an enlarged rectangle to encompass the whole object
and restricts the segmentation region
we convert multiple objects segmentation into several sub-segmentation problems
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
6 Overview of the proposed framework We built a five-view camera system for views v 01234isin
To reduce the projection error and avoid extensive computational load we select view 2 as the key view to start the segmentation process
7 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
8 Automatic initial interested objects (IIOs) extraction based on saliency model (SM)
Inspired by the work in [33] more sophisticated cues such as motion and depth are combined into our topographical SM
(a) input image (b) saliency map using depth and motion and (c) extracted IIOs
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
9 Multiple objects segmentation using graph cut
For individual object we construct a sub-graph for the pixels belonging to its ldquoObject Rectanglerdquo an enlarged rectangle to encompass the whole object
and restricts the segmentation region
we convert multiple objects segmentation into several sub-segmentation problems
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
7 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
8 Automatic initial interested objects (IIOs) extraction based on saliency model (SM)
Inspired by the work in [33] more sophisticated cues such as motion and depth are combined into our topographical SM
(a) input image (b) saliency map using depth and motion and (c) extracted IIOs
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
9 Multiple objects segmentation using graph cut
For individual object we construct a sub-graph for the pixels belonging to its ldquoObject Rectanglerdquo an enlarged rectangle to encompass the whole object
and restricts the segmentation region
we convert multiple objects segmentation into several sub-segmentation problems
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
8 Automatic initial interested objects (IIOs) extraction based on saliency model (SM)
Inspired by the work in [33] more sophisticated cues such as motion and depth are combined into our topographical SM
(a) input image (b) saliency map using depth and motion and (c) extracted IIOs
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
9 Multiple objects segmentation using graph cut
For individual object we construct a sub-graph for the pixels belonging to its ldquoObject Rectanglerdquo an enlarged rectangle to encompass the whole object
and restricts the segmentation region
we convert multiple objects segmentation into several sub-segmentation problems
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
9 Multiple objects segmentation using graph cut
For individual object we construct a sub-graph for the pixels belonging to its ldquoObject Rectanglerdquo an enlarged rectangle to encompass the whole object
and restricts the segmentation region
we convert multiple objects segmentation into several sub-segmentation problems
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
10 Objects segmentation by using graph cut
Graph cut
The general formulation of energy function
Data term smoothness term
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
11 Basic energy function
Data term evaluate the likelihood of a certain pixel p
assigned to the label fp
color (RGB) and depth information are combined
is the color distributions modeled by the Gaussian Mixture Model (GMM) is the depth modeled by the histogram modeg() denotes a Gaussian probability distributionh() is the histogram modew() is the mixture weighting coefficient is GMM component variable =d r g b is a four-dimensional feature vector for pixel p
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
12Basic energy function
Smoothness term Epq(fpfq) measures the penalty of two neighboring
pixels p and q with different labels
dist(pq) is the coordinate distance between p and q
diff(cpcq) is the average RGB color difference between p and q
βr=(2〈 (rp-rq)2〉 )-1 where 〈 middot〉 is the expectation operator for the red channel
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
13
basic energy function using (a) color (b) depth (c) combined color and depth
The result with basic energy function
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
14 The segmentation errors in the rectangles
errors occur because their color and depth information are very similar to the foreground data
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
15 Background penalty with occlusion reasoning(12)
Since we capture the same scene at different view points occluded background regions often occur around the object boundary the occluded regions have a higher probability to be the
background than the visible ones
impose a background penalty factor αbp=35
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
16 Background penalty with occlusion reasoning(22)
(a) background probability map without occlusion penalty (b) combined occlusion map (c) background probability map with occlusion penalty
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
17 The erroneous segmentations marked as ellipses
errors are mainly caused by the strong color contrast in the background comparing to the weak contrast across the ldquotruerdquo object boundary
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
18 Foreground contrast enhancement(13)
To make the color contrast representation more efficient
the average color difference is computed in the perceptually uniform L a b color space
To enhance the contrast across foregroundbackground boundary and attenuate the background contrast
adopting the motion residual information
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
19 Foreground contrast enhancement(23)
The motion residual is defined as
the smoothness term combines the L a b color and motion residual contrasts
are the motion residual of p and q
is the reconstructed image from and the motion field
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
20 Foreground contrast enhancement(33)
But combining the color contrast and motion residual contrasts will not only attenuate the background contrast but also weaken the lsquolsquotruerdquo foreground contrast
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
21 Foreground contrast enhancement(33)
we define a local color contrast to enhance the discontinuity distribution in its neighborhood calculate the local mean μ and the local variance δ of
contrast
=[26] J Wang P Bhat RA Colburn M Agrawala MF Cohen Interactive videocutout ACM Transactions on Graphics 24 (2005) 585ndash594
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
22 The result with modified energy function
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
23 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
24
projected by pixel-based disparity compensation exploits the spatial consistency
projected by pixel-based motion compensation from the mask of its previous frame enforces the temporal consistency
Multi-view video segmentation
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
25 Uncertain boundary band validation
To improve the segmentation results we construct an uncertain band along the object boundary based on an activity measure
using our graph cut algorithm to yield more accurate segmentation layers
(a) prediction mask of view 3(b) uncertain band based on the post-processing
(a) (b)
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
26 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
27 Experimental results
five-view camera system
resolution of 640 480 at frame rate of 30 frames per second (fps)
we demonstrated on two types of multi-view videos simulating different scenarios similar and low depth
different depths
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
28 Segmentation of IOs with similar and low depth
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
29 Segmentation of IOs with different depths
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
30 Comparison with othersrsquo methods(12)
compare with Kolmogorovrsquos bilayer segmentation algorithm [21] by using their test images
(a) left view (b) right view (c) result by our proposed algorithm(d) result by Kolmogorovrsquos algorithm
[21] V Kolmogorov A Criminisi A Blake G Cross C Rother Probabilistic fusion of stereo with color and contrast for bi-layer segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (9) (2006) 1480ndash1492
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
31
compare our proposed algorithm with an existing method employing multi-way cut with α-expansion [33] using our test images
Comparison with othersrsquo methods(22)
[33] WX Yang KN Ngan Unsupervised multiple object segmentation of multiview images Advanced Concepts for Intelligent Vision Systems Conference (2007) pp 178ndash189
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
32 Outline
Introduction
The proposed framework
Method Segmentation for key view
Multi-view video segmentation
Experimental results
Conclusion
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
33 Conclusion
In this paper we propose an automatic segmentation algorithm for multiple objects from multi-view video
The experiment was implemented on two representative multi-view videos
Accurate segmentation results with good visual quality and subjective comparison with othersrsquo methods attest to the efficiency and robustness of our proposed algorithm
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-
34
Thanks for your listening
- Multi-view video based multiple objects segmentation using grap
- Outline
- Outline (2)
- Introduction
- Outline (3)
- Overview of the proposed framework
- Outline (4)
- Automatic initial interested objects (IIOs) extraction based on
- Multiple objects segmentation using graph cut
- Objects segmentation by using graph cut
- Basic energy function
- Basic energy function (2)
- The result with basic energy function
- The segmentation errors in the rectangles
- Background penalty with occlusion reasoning(12)
- Background penalty with occlusion reasoning(22)
- The erroneous segmentations marked as ellipses
- Foreground contrast enhancement(13)
- Foreground contrast enhancement(23)
- Foreground contrast enhancement(33)
- Foreground contrast enhancement(33) (2)
- The result with modified energy function
- Outline (5)
- Multi-view video segmentation
- Uncertain boundary band validation
- Outline (6)
- Experimental results
- Segmentation of IOs with similar and low depth
- Segmentation of IOs with different depths
- Comparison with othersrsquo methods(12)
- Comparison with othersrsquo methods(22)
- Outline (7)
- Conclusion
- Thanks for your listening
-