Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video...
-
Upload
brice-glenn -
Category
Documents
-
view
232 -
download
0
Transcript of Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video...
![Page 1: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/1.jpg)
Temporal Video Bound-aries
Computer Science EngineeringLee Sang Seon
![Page 2: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/2.jpg)
WhyTemporal Video Boundaries
Techniqueis useful in the
Video content analysis?
![Page 3: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/3.jpg)
Index
Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries Conclusion Q & A
![Page 4: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/4.jpg)
Introduction
Brief definition of Temporal Video Boundary technique
→ Examine the temporal boundary problem at different levels of video content structure analysis
Why we need Temporal Video Boundary technique?
Show example
![Page 5: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/5.jpg)
Example : Oscar awards
Insufficient metadata
opening
ending
![Page 6: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/6.jpg)
Example : Oscar awards
Detailed metadata
opening
ending
actor
winners
awards
ending
![Page 7: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/7.jpg)
Basic notions - modali-ties Video contains three types of modalities (i) Visual (ii) Audio (iii) Textual
Each modality has three levels(i) low-level (ii) mid -level (iii) high-level→ levels describe the amount of details described in each modality in terms of granularity and ab-straction
![Page 8: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/8.jpg)
Basic notions - modali-ties For each modality and for each level there if
a set of attributes. These can be formalized as vectors:
![Page 9: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/9.jpg)
Basic notions - modali-ties Adding to this, given a set of vectors
→ their average value denote the vector
![Page 10: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/10.jpg)
Basic notions - method Local method→ the difference is computed between con-
secutive frames
Global method→ the difference if computed over a series of
frames
![Page 11: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/11.jpg)
Micro-Boundaries
Definition Boundaries associated to the smallest video
units for which a given attribute is constant or slowly varying
The attribute can be any feature in the visual, audio, or text domain
![Page 12: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/12.jpg)
Example
![Page 13: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/13.jpg)
Make family histogram
Data structure that represents the color in-formation of a family of frames.
Set of frames that exhibits uniform features
= Frame histogram
![Page 14: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/14.jpg)
Histogram difference measures Histogram difference using L1 metrics
Bin-wise histogram intersection
Total number of color bins used
Histogram of previous frame
Histogram of current frame
![Page 15: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/15.jpg)
Merging of family his-tograms
![Page 16: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/16.jpg)
Multiple ways to compare and merge families - contiguity & memory
1. Contiguous with zero memory → A new frame histogram is compared with
previous frame histogram
2. Contiguous with limited memory→ A new frame histogram is compared with
previous family histogram
![Page 17: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/17.jpg)
Multiple ways to compare and merge families - contiguity & memory
3. Non contiguous with unlimited memory → A new frame histogram is compared with all
previous family histograms within the same video.
4. Hybrid→ First a new frame histogram is compared using
the contiguous frames and then generated family histograms are merged using non con-tiguous case.
![Page 18: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/18.jpg)
Compare different Histogram difference measures
![Page 19: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/19.jpg)
Macro-Boundaries
Definition Boundaries between collections of video micro-
segments that are clearly identifiable organic parts of an event defining a structural (action) or thematic (story) unit
Video : collection of stories that may or may not be interconnected
→ Macro-Boundaries detection= Segmenting stories
textual cues
audio cuesvisual cues
![Page 20: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/20.jpg)
Two types of uniform segment detection
Unimodal segment detection A video segment exhibits same characteristic
over a period of time
Multimodal segment detection A video segment exhibits a certain characteris-
tic taking into account attributes from different modalities
![Page 21: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/21.jpg)
Single Modality Segmen-taion
Partition a continuous bit-stream of audio data into non-
overlapping segments
Classification
Seven mid-level audio cate-gories
Using low-level audio features
Audio segmen-tation & classifi-
cationText transcript
Extracted from either the closed captions or speech-to-
text conversion
Segmented and categorized with respect to a predefined
topic list
Frequency-of-word-occurrence metric is used
![Page 22: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/22.jpg)
Multimodal Segmentaion
Pre-merging Steps
Uniform seg-ment detection
Intra-modal segment clus-
tering
Attribute tem-plate determi-
nation
Dominant at-tribute deter-
mination
Template ap-plication
Descent Meth-ods
Goal :Create macro-bound-
aries that are more ac-curate than the bound-aries produced by indi-
vidual modalities.
![Page 23: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/23.jpg)
Descent MethodsText seg-
ment
Audio segment
Video segment
![Page 24: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/24.jpg)
Single descent Method
Single descent with intersecting
union
Single descent with intersec-
tion
Single descent with secondary
voting attributes
Single descent with conditional
union
![Page 25: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/25.jpg)
Mega-Boundaries
Definition Boundaries between collections of macro-seg-
ments that exhibit different structural and fea-ture consistency (e.g. different genres)
Example Commercial detection method
![Page 26: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/26.jpg)
Trigger & Verifiers Model
Features that can aid in determining the location of the commercial break
Triggers
Features that can determine the boundaries of the commercial break
Verifiers
![Page 27: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/27.jpg)
Black frames
Time interval be-tween detected
black frames as trig-gers
Used as verifiers
Letterbox change
High cut rate(= low cut distance)
![Page 28: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/28.jpg)
Bayesian Belief Network Modelstart
![Page 29: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/29.jpg)
Genetic Algorithms
![Page 30: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/30.jpg)
Conclusion
Type of bound-aries Methods Example
Micro-boundaries Frame & Family histogram comparing and merging
Visual scene segmenta-tion
Macro-boundaries Single modality segmenta-tion
&Multimodal segmentation
Multimodal story segmen-tation
Mega-boundaries Trigger & Verifier Commercial detection
![Page 31: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/31.jpg)
Whenever metadata is availableor unavailable,
we can segment video by using this technique that
categorized three types
![Page 32: Computer Science Engineering Lee Sang Seon. Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries.](https://reader038.fdocuments.net/reader038/viewer/2022102718/56649cf35503460f949c1214/html5/thumbnails/32.jpg)
&Thank you!
Q & A