Today’s Topics Image and video processing Image and video applications.
-
Upload
emery-harrell -
Category
Documents
-
view
242 -
download
1
Transcript of Today’s Topics Image and video processing Image and video applications.
![Page 1: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/1.jpg)
Today’s Topics
• Image and video processing• Image and video applications
![Page 2: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/2.jpg)
Image Processing: Color• Color histograms – how
much of each color is in image– Probability of a pixel in the
image being a particular color• Color correlograms – how
close colors are to each other in image– Probability of finding a pixel
of a particular color at a specific distance from a pixel of a known color
![Page 3: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/3.jpg)
Image/Video Processing: Subdividing
• Region subdivision– Sometimes we subdivide
images into regions– Spread observed features
at edges for more continuous model
• Temporal subdivision– Video is subdivided into
segments– Spread features into
neighboring segments
![Page 4: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/4.jpg)
4
Image Processing: Foreground Background Separation
• Background Modeling– Convert to greyscale– Dynamic model (to cope with changes in signer
body position and lighting)• BPt = .96 * BP(t-1) + .04 P
• Foreground object detection– Pixels different from background model by more
than a threshold are foreground pixels– Spatial filter removes regions of foreground pixels
smaller than a minimum threshold
• Face location to determine position of foreground relative to the face
• Videos without a single main face are not considered as potential SL videos
![Page 5: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/5.jpg)
Image Processing: Other Features
• Edge detection– Sobel filter
• Object and Face detection– Skintone models
• Face recognition• Open Source
Computer Vision (OpenCV)
![Page 6: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/6.jpg)
Example Applications
• MediaGlow– Clustering and selecting images from collection
• DOTS– Interpreting surveillance video
• HyperHitchcock– Authoring and viewing interactive video
![Page 7: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/7.jpg)
MediaGLOW: The Concept
• Task and Opportunity– Many tasks involving photo sharing or publication involve grouping a
candidate photos into categories and selecting from among these groups.
– There are many different reasons to group photos: based on event, location, subject matter, visual characteristics, etc.
– There is the potential for systems to observe earlier user activity and expression to aid their later activity.
![Page 8: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/8.jpg)
MediaGLOW: Interpreting User Action
• Evolving Notion of Similarity via User Expression– Photos presented in a graph-based
workspace with “springs” between each pair of photos.
– Lengths of springs is initially based on a default distance metric based on their time, location, tags, or visual features.
– Users can pin photos in place and create piles of photos.
– Distance metric to piles change as new members are added, resulting in the dynamic layout of unpinned photos in the workspace.
![Page 9: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/9.jpg)
MediaGLOW: Multi-faceted Visualizationof Photo Collections
• System Expression through Neighborhoods– Piles have neighborhood for photos
that are similar to the pile based on the pile’s unique distance metric.
– Photos in a neighborhood are only connected to other photos in the neighborhood, enabling piles to be moved independent of each other.
– Lingering over a pile visualizes how similar other piles are to that pile, indicating system ambiguity in categories.
![Page 10: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/10.jpg)
DOTS: Supporting Use of Surveillance Video
• The problem– Number and size of
surveillance systems are increasing but human attention is limiting factor
• Approach– Provide summaries
of action– Build interfaces
knowing limits of automation
– https://www.youtube.com/watch?v=9wVAVm8-bQ8
![Page 11: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/11.jpg)
DOTS: The Main Interface
• Components– Rotating camera
bank with activity graphs
– Mixed-initiative main viewer
– Map with tracking data
– Timeline with automatic events
![Page 12: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/12.jpg)
DOTS: Tracking Layout
• Difficulty in tracking is that camera views are often similar
• Tracking layout places cameras around the main viewer to aid tracking
• Study showed significant improvement in tracking success over traditional viewer
• In either layout, map can be used to find activity near a location and time.
![Page 13: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/13.jpg)
HyperHitchcock: Interactive Video
• Issue– Vision: Seamlessly interact
with charactersin the show
– Reality: Difficult to author even simpleinteractive videos
• Today, video is included within pages of content but links between playing videos are not common.
![Page 14: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/14.jpg)
Support for Hypervideo Authoring
• Links in video can lead to other video segments– Short main video with branches providing additional detail– Hyperlinks to branches just like in Web pages– Making of a scene in a movie, biography of an actor,
different camera angle• General hypervideo difficult to author
– Simple hypervideo format with only a single active link• Novel approach: use automatic video analysis, create
an easy-to-use interface, and support simple hypervideo format
![Page 15: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/15.jpg)
Uses of Hypervideo
• Hypervideo well-suited for training video– Overview of topic with links to more detail– Viewers can choose video content based on their prior
knowledge and current task• Home video is more enjoyable if viewers can select
content– Only a small portion of the video is really “usable”– Difficult to watch long home videos – Customize viewing experience for different viewers– More detail on sports events for some family members,
more scenes with children for others
![Page 16: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/16.jpg)
Detail-on-demand Hypervideo
• General hypervideo– Links from objects in video to other video– Requires object tracking– Requires interface for indicating and selecting from
multiple links
• Detail-on-demand video– Single link from any video segment– No anchor regions to simplify viewing and authoring– https://www.youtube.com/watch?v=LaXZuLA-68g
![Page 17: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/17.jpg)
Hierarchical Video with Links
• Video sequences are represented as a containment hierarchy of video elements– Elements are video clips or composites grouping other video elements– Elements are played in sequence
• Each element can be link anchor or link destination• Anchor for innermost element is available while element is
playing• After link destination
video is played,play-back continuesat the link anchor
![Page 18: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/18.jpg)
Detail-on-demand Links• Any video clip or composite can be link anchor or link destination• Optional link offsets into destination• Links have labels• Link return behaviors control the purpose of the link
– Play from where the viewer left the video– Play from the end of the source anchor sequence– Play from beginning of the
source anchor sequence– Stop playback
• Different behaviors fordestination completion oraborted playback
![Page 19: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/19.jpg)
Hyper-Hitchcock Editor & Player Goals
• Hyper-Hitchcock editor designed to– Reduce cost of producing
interactive video– Make resulting video useful
to wider audience
• Simple video player to simulate DVD viewing experience
![Page 20: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/20.jpg)
Hyper-Hitchcock Editor
• Hyper-Hitchcock evolved from Hitchcock video editor• Video clips grouped in piles by similarity (e.g., recording time)• Workspace to arrange clips
– Resize keyframes to trim clips– Clips ordered as horizontal
or vertical lists– Place links between clips– Group clips into composites
• Tree view to visualizecontainment hierarchy ofcomposites
![Page 21: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/21.jpg)
Determining Suitable Video Clips• Unsuitability score
– Single score for video features such as camera motion and brightness– Estimate camera pan by shifting frames against each other– Require minimum brightness
• Determine clip boundaries– Select clips that fall in “valleys” between unsuitability peaks – Look for areas completely above unsuitability threshold
• Peak is clip boundary candidate– Enforce the minimum length requirement
• Trimming clips– Select portion of clip with minimal area under the curve– Expand the area for longer requested portions
![Page 22: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/22.jpg)
Selected Clip Portions
![Page 23: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/23.jpg)
Trimming Clips in the Workspace
• Best five seconds of clip selected by default• Resizing keyframe changes length of clip
– Picks the best portion around initial five-second portion– Start and end can jump to sentence boundary silence
• Clip start and/or end can be locked in timeline• Locked ends can be dragged• Audio energy
visualized in timelineto spot words andsentences
![Page 24: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/24.jpg)
Video clips to be grouped into a composite(keyframe area proportional to clip length)
Composite visualizations
cross(area error minimized)
sliding dividers(area proportional to clip length)
Visualizing Video Composites
![Page 25: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/25.jpg)
Attaching Links to Clips and Composites
• Link anchors and destinations can be clips, composites, or elements inside composites
• Color-coding and position indicates link attachment in workspace
Links in and out of compositeBlue: attached to compositeRed: attached to elementDashed: between composite and element
![Page 26: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/26.jpg)
Hypervideo Player• Video player with controls for following and returning
from links• Several improvements based on user feedback
– First version indicated linksin timeline and showed thelabel for the active link
– Next version showedlabels in timeline
– Current version includeskeyframes for active linkand for link history
• User study suggestsfurther improvements
![Page 27: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/27.jpg)
Navigation Aids in Hypervideo Player
• Keyframe list for navigation history– Shows followed links with more remote
navigation indicated by size of keyframe– Label of followed link as video caption
• Link indicator in timeline (in blue)– Link labels and keyframes in timeline– Keyframe grows, label completes, and
link recolored when active
![Page 28: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/28.jpg)
Impressions from Users• Pilot editing study with two participants
– Subjects own video• report on a trip to Japan• family outing to mountain bike race
– Few problems but link return behaviorwas confusing for one participant
• Study of hypervideo player (6 subjects)– Plumbing training video re-authored
as hypervideo– All participants were able to find answers
quickly in the hypervideo (9 - 20 minutes in 60-minute video)– Navigation through video can be confusing. As player was altered
to better support navigation, playback becomes less like video.
![Page 29: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/29.jpg)
Generating Hypervideo Summaries
• Locating content in video is time consuming.– Much effort into generating “good” video
summaries.– But what is good for one task is not good for
another.• Generate hypervideo summaries that allow
users to determine the level of detail viewed.
![Page 30: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/30.jpg)
Process for Generating Summaries
• Determine number of summary levels– Based on length of source video
• Select clips to include in each summary level– Clips found by subdividing takes by camera motion– Select clips via clip distribution, take distribution, or best-
first algorithms
• Add links between summary levels– Group clips by takes– Links between clips from the same take using simple take-
to-take or take-to-take with offsets algorithms
![Page 31: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/31.jpg)
• Selecting n clips out of m candidate clips– Evenly distribute in among candidate clips
– Selects more clips from takes with many recognized clips.– Good when take includes more than one topic/activity.– Bad when lots of clips are for a single topic/activity.
Clip Distribution Selection
![Page 32: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/32.jpg)
Evenly distribute in time and takes Divide video duration into n time segments Select clip nearest center of segment in take not already
represented.
Bad when take includes more than one topic/activity. Good when lots of clips are for a single topic/activity.
Take Distribution for Clip Selection
![Page 33: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/33.jpg)
Best-first Clip Selection
Assumes human or automated ordering of value of clipsSimply selects n best clips
Good in cases of edited video – not currently applicable for unedited video.Best can be introductory materialBest can be highlights of material
![Page 34: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/34.jpg)
Links in Video Summary• All clips from a take are grouped into composite• Single clip from take or take composite is used as link
anchor and destination• Simple take-to-take algorithm
– Links take composite to take composite– Best when single activity divided into multiple clips.
• Take-to-take with offsets algorithm– Each clip from take links to take composite in next level
with offset to temporally closest clip. – Best when clips portray
multiple activities in take.• Complete takes for
entire video
![Page 35: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/35.jpg)
Automatically Generated Summary
0:33
3:35
14:44
60:42
A four-level hypervideo summary of a one-hour source video. Lower levels provide more detail.
Time
![Page 36: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/36.jpg)
Hypervideo in Practice
![Page 37: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/37.jpg)
Experience with Use
• Nine hypervideos each authored by 1-2 students in a Computers and New Media class
• Recorded up to 1 hour using a DV camera then authored hypervideo in HH
• Students had about 1 week for authoring activity.
![Page 38: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/38.jpg)
Riding Down University DriveMost directly maps geographic structure
Links are choices to stop in at sites along road.
![Page 39: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/39.jpg)
“In Danger” BuildingsVisits to buildings identified for demolition on TAMU
campus.
Links are to seeing interior and details.
![Page 40: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/40.jpg)
Perspectives on Bridges
Shows bridges between College Station and Austin.
Presents different perspectives (roadway, construction, wildlife) in order.
Links are to seeing different perspective.
![Page 41: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/41.jpg)
International Dance FestivalShows bits of performances.
Brief “how-to use hypervideo player” at start.
Links are to seeing more and to next performance.
![Page 42: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/42.jpg)
Game WalkthroughShows what happens while playing game.
Links represent choice points in game.
![Page 43: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/43.jpg)
Music Hypervideopublic-domain video, remixed audio
![Page 44: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/44.jpg)
Home Hypervideoidiosyncratic structure
![Page 45: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/45.jpg)
Roles of Links
• Detail Links• Prerequisite Links• Related Information
Links• Alternate View Links• Action Choice Links
![Page 46: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/46.jpg)
Summary
• Hyper-Hitchcock used to author documentary, how-to, music, and home hypervideos.
• Links in hypervideos used for: details, prerequisites, alternate views, action choices, and related information.
• Structures in hypervideo were impacted by the inclusion of return behaviors.
![Page 47: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/47.jpg)
Conclusions
• Detail-on-demand video well-suited for training and home video– Simple interaction style appropriate for
DVD-player interfaces– Enables wide range of authors due to emphasis on ease of learning
and use over richness of interaction
• Hypervideo summaries – Remove need for single context-free video summary– Multiple clip selection and link generation algorithms– Act as starting point for human-authored summary
• Hyper-Hitchcock used to author documentary, how-to, music, and home hypervideos
• Links in hypervideos used for: details, prerequisites, alternate views, action choices, and related information
![Page 48: Today’s Topics Image and video processing Image and video applications.](https://reader030.fdocuments.net/reader030/viewer/2022012916/56649e755503460f94b75d9b/html5/thumbnails/48.jpg)
Today’s Topics
• Image and video processing– Color-oriented representations– Region and temporal segmentation– Foreground-background separation– Edge and face detection
• Image and video applications– MediaGlow – image selection– DOTS – surveillance– HyperHitchcock – interactive video