Hyper-Hitchcock:Hyper-Hitchcock: Authoring Interactive Videos Authoring Interactive Videos and Generating Interactive and Generating Interactive

SummariesSummaries

Frank ShipmanFrank ShipmanTexas A&M UniversityTexas A&M University

Andreas GirgensohnAndreas GirgensohnLynn WilcoxLynn Wilcox

FX Palo Alto Laboratory, Inc.FX Palo Alto Laboratory, Inc.

OutlineOutline

IntroductionIntroductionDetail-on-demand videoDetail-on-demand videoHyper-Hitchcock hypervideo editorHyper-Hitchcock hypervideo editorGenerating hypervideo summariesGenerating hypervideo summariesExperience with AuthoringExperience with AuthoringConclusionsConclusions

Computational MediaComputational Media

Computation-based mediums of Computation-based mediums of communication are growingcommunication are growing E-mail, IM, Blogs, MMORPG, …E-mail, IM, Blogs, MMORPG, … Text, images, animation, …Text, images, animation, … TV/DVD/PC merger (Sony PS3, TV/DVD/PC merger (Sony PS3,

Media Center PCs)Media Center PCs) Interactive videoInteractive video

Vision: Seamlessly interact with charactersVision: Seamlessly interact with charactersin the showin the show

Reality: Difficult to author even simpleReality: Difficult to author even simpleinteractive videosinteractive videos

Support for Hypervideo Support for Hypervideo AuthoringAuthoring

Links in video can lead to other video Links in video can lead to other video segmentssegments Short main video with branches providing Short main video with branches providing

additional detailadditional detail Hyperlinks to branches just like in Web pagesHyperlinks to branches just like in Web pages Making of a scene in a movie, biography of an Making of a scene in a movie, biography of an

actor, different camera angleactor, different camera angle General hypervideo difficult to authorGeneral hypervideo difficult to author

Simple hypervideo format with only a single Simple hypervideo format with only a single active linkactive link

Novel approach: use automatic video Novel approach: use automatic video analysis, create an easy-to-use interface, analysis, create an easy-to-use interface, and support simple hypervideo formatand support simple hypervideo format

Uses of HypervideoUses of Hypervideo

Hypervideo well-suited for training videoHypervideo well-suited for training video Overview of topic with links to more detailOverview of topic with links to more detail Viewers can choose video content based on Viewers can choose video content based on

their prior knowledge and current tasktheir prior knowledge and current task Home video is more enjoyable if viewers Home video is more enjoyable if viewers

can select contentcan select content Only a small portion of the video is really Only a small portion of the video is really

“usable”“usable” Difficult to watch long home videos Difficult to watch long home videos Customize viewing experience for different Customize viewing experience for different

viewersviewers More detail on sports events for some family More detail on sports events for some family

members, more scenes with children for othersmembers, more scenes with children for others

Detail-on-demand HypervideoDetail-on-demand Hypervideo

General hypervideoGeneral hypervideo Links from objects in video to other videoLinks from objects in video to other video Requires object trackingRequires object tracking Requires interface for indicating and selecting Requires interface for indicating and selecting

from multiple linksfrom multiple links Detail-on-demand videoDetail-on-demand video

Single link from any video segmentSingle link from any video segment No anchor regions to simplify viewing and No anchor regions to simplify viewing and

authoringauthoring

Hierarchical Video with LinksHierarchical Video with Links

Video sequences are represented as a containment Video sequences are represented as a containment hierarchy of video elementshierarchy of video elements Elements are video clips or composites grouping other Elements are video clips or composites grouping other

video elementsvideo elements Elements are played in sequenceElements are played in sequence

Each element can be link anchor or link destinationEach element can be link anchor or link destination Anchor for innermost element is available while Anchor for innermost element is available while

element is playingelement is playing After link destinationAfter link destination

video is played,video is played,play-back continuesplay-back continuesat the link anchorat the link anchor

Detail-on-demand LinksDetail-on-demand Links

Any video clip or composite can be link anchor or link Any video clip or composite can be link anchor or link destinationdestination

Optional link offsets into destinationOptional link offsets into destination Links have labelsLinks have labels Link return behaviors control the purpose of the linkLink return behaviors control the purpose of the link

Play from where the viewer left the videoPlay from where the viewer left the video Play from the end of the source anchor sequencePlay from the end of the source anchor sequence Play from beginning of thePlay from beginning of the

source anchor sequencesource anchor sequence Stop playbackStop playback

Different behaviors forDifferent behaviors fordestination completion ordestination completion oraborted playbackaborted playback

Hyper-Hitchcock Editor & PlayerHyper-Hitchcock Editor & Player GoalsGoals

Hyper-Hitchcock Hyper-Hitchcock editor designed toeditor designed to Reduce cost of Reduce cost of

producing interactive producing interactive videovideo

Make resulting video Make resulting video useful to wider useful to wider audienceaudience

Simple video player to Simple video player to simulate DVD viewing simulate DVD viewing experienceexperience

Hyper-Hitchcock EditorHyper-Hitchcock Editor

Hyper-Hitchcock evolved from Hitchcock video editorHyper-Hitchcock evolved from Hitchcock video editor Video clips grouped in piles by similarity (e.g., Video clips grouped in piles by similarity (e.g.,

recording time)recording time) Workspace to arrange clipsWorkspace to arrange clips

Resize keyframes to trim clipsResize keyframes to trim clips Clips ordered as horizontalClips ordered as horizontal

or vertical listsor vertical lists Place links between clipsPlace links between clips Group clips into compositesGroup clips into composites

Tree view to visualizeTree view to visualizecontainment hierarchy ofcontainment hierarchy ofcompositescomposites

Determining Suitable Video Determining Suitable Video ClipsClips

Unsuitability scoreUnsuitability score Single score for video features such as camera motion and Single score for video features such as camera motion and

brightnessbrightness Estimate camera pan by shifting frames against each otherEstimate camera pan by shifting frames against each other Require minimum brightnessRequire minimum brightness

Determine clip boundariesDetermine clip boundaries Select clips that fall in “valleys” between unsuitability peaks Select clips that fall in “valleys” between unsuitability peaks Look for areas completely above unsuitability thresholdLook for areas completely above unsuitability threshold

Peak is clip boundary candidatePeak is clip boundary candidate Enforce the minimum length requirementEnforce the minimum length requirement

Trimming clipsTrimming clips Select portion of clip with minimal area under the curveSelect portion of clip with minimal area under the curve Expand the area for longer requested portionsExpand the area for longer requested portions

Selected Clip PortionsSelected Clip Portions

Trimming Clips in the Trimming Clips in the WorkspaceWorkspace

Best five seconds of clip selected by defaultBest five seconds of clip selected by default Resizing keyframe changes length of clipResizing keyframe changes length of clip

Picks the best portion around initial five-second portionPicks the best portion around initial five-second portion Start and end can jump to sentence boundary silenceStart and end can jump to sentence boundary silence

Clip start and/or end can be locked in timelineClip start and/or end can be locked in timeline Locked ends can be draggedLocked ends can be dragged Audio energyAudio energy

visualized in timelinevisualized in timelineto spot words andto spot words andsentencessentences

Video clips to be grouped into a composite(keyframe area proportional to clip length)

Composite visualizations

cross(area error minimized)

sliding dividers(area proportional to clip length)

Visualizing Video CompositesVisualizing Video Composites

Attaching Links to Clips and Attaching Links to Clips and CompositesComposites

Link anchors and Link anchors and destinations can be clips, destinations can be clips, composites, or elements composites, or elements inside compositesinside composites

Color-coding and position Color-coding and position indicates link attachment indicates link attachment in workspacein workspace

Links in and out of compositeBlue: attached to compositeRed: attached to elementDashed: between composite and element

Hypervideo PlayerHypervideo Player Video player with controls for following and Video player with controls for following and

returning from linksreturning from links Several improvements based on user Several improvements based on user

feedbackfeedback First version indicated linksFirst version indicated links

in timeline and showed thein timeline and showed thelabel for the active linklabel for the active link

Next version showedNext version showedlabels in timelinelabels in timeline

Current version includesCurrent version includeskeyframes for active linkkeyframes for active linkand for link historyand for link history

User study suggestsUser study suggestsfurther improvementsfurther improvements

Navigation Aids in Hypervideo Navigation Aids in Hypervideo PlayerPlayer

Keyframe list for navigation historyKeyframe list for navigation history Shows followed links with more Shows followed links with more

remote navigation indicated by size of remote navigation indicated by size of keyframekeyframe

Label of followed link as video captionLabel of followed link as video caption Link indicator in timeline (in blue)Link indicator in timeline (in blue)

Link labels and keyframes in timelineLink labels and keyframes in timeline Keyframe grows, label completes, and Keyframe grows, label completes, and

link recolored link recolored when activewhen active

Impressions from UsersImpressions from Users Pilot editing study with two participantsPilot editing study with two participants

Subjects own videoSubjects own video report on a trip to Japanreport on a trip to Japan family outing to mountain bike racefamily outing to mountain bike race

Few problems but link return behaviorFew problems but link return behaviorwas confusing for one participantwas confusing for one participant

Study of hypervideo player (6 subjects)Study of hypervideo player (6 subjects) Plumbing training video re-authored Plumbing training video re-authored

as hypervideoas hypervideo All participants were able to find answersAll participants were able to find answers

quickly in the hypervideo (9 - 20 minutes in 60-quickly in the hypervideo (9 - 20 minutes in 60-minute video)minute video)

Navigation through video can be confusing. As player Navigation through video can be confusing. As player was altered to better support navigation, playback was altered to better support navigation, playback becomes less like video.becomes less like video.

Generating Hypervideo Generating Hypervideo SummariesSummaries

Locating content in video is time Locating content in video is time consuming.consuming.Much effort into generating “good” Much effort into generating “good”

video summaries.video summaries.But what is good for one task is not But what is good for one task is not

good for another.good for another.Generate hypervideo summaries that Generate hypervideo summaries that

allow users to determine the level of allow users to determine the level of detail viewed.detail viewed.

Process for Generating Process for Generating SummariesSummaries

Determine number of summary levelsDetermine number of summary levels Based on length of source videoBased on length of source video

Select clips to include in each summary levelSelect clips to include in each summary level Clips found by subdividing takes by camera motionClips found by subdividing takes by camera motion Select clips via clip distribution, take distribution, Select clips via clip distribution, take distribution,

or best-first algorithmsor best-first algorithms Add links between summary levelsAdd links between summary levels

Group clips by takesGroup clips by takes Links between clips from the same take using Links between clips from the same take using

simple take-to-take or take-to-take with offsets simple take-to-take or take-to-take with offsets algorithmsalgorithms

Selecting n clips out of m candidate clipsSelecting n clips out of m candidate clips Evenly distribute in among candidate clipsEvenly distribute in among candidate clips

Selects more clips from takes with many Selects more clips from takes with many recognized clips.recognized clips.

Good when take includes more than one Good when take includes more than one topic/activity.topic/activity.

Bad when lots of clips are for a single Bad when lots of clips are for a single topic/activity.topic/activity.

Clip Distribution SelectionClip Distribution Selection

Evenly distribute in time and takesEvenly distribute in time and takes Divide video duration into n time segmentsDivide video duration into n time segments Select clip nearest center of segment in take not Select clip nearest center of segment in take not

already represented.already represented.

Bad when take includes more than one topic/activity.Bad when take includes more than one topic/activity. Good when lots of clips are for a single topic/activity.Good when lots of clips are for a single topic/activity.

Take Distribution for Clip Take Distribution for Clip SelectionSelection

Best-first Clip SelectionBest-first Clip Selection

Assumes human or automated Assumes human or automated ordering of value of clipsordering of value of clipsSimply selects n best clipsSimply selects n best clips

Good in cases of edited video – not Good in cases of edited video – not currently applicable for unedited currently applicable for unedited video.video.Best can be introductory materialBest can be introductory materialBest can be highlights of materialBest can be highlights of material

Links in Video SummaryLinks in Video Summary All clips from a take are grouped into All clips from a take are grouped into

compositecomposite Single clip from take or take composite is Single clip from take or take composite is

used as link anchor and destinationused as link anchor and destination Simple take-to-take algorithmSimple take-to-take algorithm

Links take composite to take compositeLinks take composite to take composite Best when single activity divided into multiple clips.Best when single activity divided into multiple clips.

Take-to-take with offsets algorithmTake-to-take with offsets algorithm Each clip from take links to take composite in next Each clip from take links to take composite in next

level with offset to temporally closest clip. level with offset to temporally closest clip. Best when clips portray Best when clips portray

multiple activities in take.multiple activities in take. Complete takes for Complete takes for

entire videoentire video

Automatically Generated Automatically Generated SummarySummary

0:33

3:35

14:44

60:42

A four-level hypervideo summary of a one-hour source video. Lower levels provide more detail.

Time

Future Systems ResearchFuture Systems Research

MPEG-4 players support hypervideoMPEG-4 players support hypervideo No good authoring toolsNo good authoring tools Demand for easy-to-use tools for quickly editing Demand for easy-to-use tools for quickly editing

hypervideohypervideo Research problems for hypervideo playerResearch problems for hypervideo player

Navigation: easy to get lost; how to visualize placeNavigation: easy to get lost; how to visualize place Viewing: user interface for TV remote control; other input Viewing: user interface for TV remote control; other input

devices such as data glovesdevices such as data gloves Keyframes for links represent a first step towards solutionsKeyframes for links represent a first step towards solutions

Research problems for hypervideo summaries Research problems for hypervideo summaries Different summaries for different genres (e.g., home video, Different summaries for different genres (e.g., home video,

produced training video)produced training video) Clip selection and link placement dependent on the genreClip selection and link placement dependent on the genre

Hypervideo in PracticeHypervideo in Practice

Experience with UseExperience with Use

Nine hypervideos each authored by Nine hypervideos each authored by 1-2 students in a Computers and 1-2 students in a Computers and New Media classNew Media class

Recorded up to 1 hour using a DV Recorded up to 1 hour using a DV camera then authored hypervideo in camera then authored hypervideo in HHHH

Students had about 1 week for Students had about 1 week for authoring activity.authoring activity.

Riding Down University Riding Down University DriveDrive

Most directly maps geographic structureMost directly maps geographic structure

Links are choices to stop in at sites along road.Links are choices to stop in at sites along road.

Perspectives on BridgesPerspectives on Bridges

Shows bridges between College Station and Austin.Shows bridges between College Station and Austin.

Presents different perspectives (roadway, Presents different perspectives (roadway, construction, wildlife) in order.construction, wildlife) in order.

Links are to seeing different perspective.Links are to seeing different perspective.

““In Danger” BuildingsIn Danger” Buildings

Visits to buildings identified for demolition on Visits to buildings identified for demolition on TAMU campus.TAMU campus.

Links are to seeing interior and details.Links are to seeing interior and details.

International Dance FestivalInternational Dance Festival

Shows bits of performances.Shows bits of performances.

Brief “how-to use hypervideo player” at start.Brief “how-to use hypervideo player” at start.

Links are to seeing more and to next performance.Links are to seeing more and to next performance.

Game WalkthroughGame Walkthrough

Shows what happens while playing game.Shows what happens while playing game.

Links represent choice points in game.Links represent choice points in game.

Preparing for the In-LawsPreparing for the In-Laws

How-to for cleaning and cooking, includes How-to for cleaning and cooking, includes constant audio commentary.constant audio commentary.

Links are to greater detail.Links are to greater detail.

How To Make a SandwichHow To Make a Sandwich

Presents choices in sandwich creation.Presents choices in sandwich creation.

Includes video of options.Includes video of options.

Links are to select another option.Links are to select another option.

Music HypervideoMusic Hypervideo

public-domain video, remixed audiopublic-domain video, remixed audio

Home HypervideoHome Hypervideo

idiosyncratic structureidiosyncratic structure

Feedback on Hyper-Feedback on Hyper-HitchcockHitchcock

Evaluation of Hyper-Evaluation of Hyper-HitchcockHitchcock

Easy to use hypervideo player, to work Easy to use hypervideo player, to work with video clips, to create composites.with video clips, to create composites.

Not as easy to manipulate composites Not as easy to manipulate composites and create links.and create links.

Spatial parser correctly interpreted Spatial parser correctly interpreted composites.composites.

Link labels were valued by many.Link labels were valued by many. Audio energy graph was only valued by Audio energy graph was only valued by

a few.a few. Automatic segmentation of source video Automatic segmentation of source video

into clips and automatic selection of into clips and automatic selection of in/out points did not help.in/out points did not help.

Structuring ProblemsStructuring Problems

Authoring the two separate Authoring the two separate structures confused some studentsstructures confused some studentsComposition of video clips into linear Composition of video clips into linear

videosvideosLinks between linear videosLinks between linear videos

Need for greater explanation of how Need for greater explanation of how the two structures interactthe two structures interact

Suggested FeaturesSuggested Features

Ability to have more than one link Ability to have more than one link active at a time.active at a time.

Ability to specify for video to stop Ability to specify for video to stop playing at decision point.playing at decision point.

Separate audio editorSeparate audio editorText overlays and video effectsText overlays and video effects

Roles of LinksRoles of Links

Detail LinksDetail Links Prerequisite LinksPrerequisite Links Related Related

Information LinksInformation Links Alternate View Alternate View

LinksLinks Action Choice LinksAction Choice Links

Hypervideos Mimicking Hypervideos Mimicking HypertextHypertext

Detail links resulted in sieves.Detail links resulted in sieves.Some sieves included links from Some sieves included links from

one detailed section to the next one detailed section to the next (shortcuts).(shortcuts).

Related information links also Related information links also created tangles.created tangles.

Alternate view links likely to Alternate view links likely to generate mirrorworld or generate mirrorworld or counterpoint structures.counterpoint structures.

Hypervideo DifferencesHypervideo Differences

Split/joins and mirrorworlds used to Split/joins and mirrorworlds used to merge alternate courses of actions or merge alternate courses of actions or perspectives.perspectives.

Return behaviors used to cause Return behaviors used to cause merging paths, thus the structure merging paths, thus the structure looks like a sieve.looks like a sieve.

SummarySummary

Hyper-Hitchcock used to author Hyper-Hitchcock used to author documentary, how-to, music, and documentary, how-to, music, and home hypervideos.home hypervideos.

Links in hypervideos used for: Links in hypervideos used for: details, prerequisites, alternate details, prerequisites, alternate views, action choices, and related views, action choices, and related information.information.

Structures in hypervideo were Structures in hypervideo were impacted by the inclusion of return impacted by the inclusion of return behaviors.behaviors.

ConclusionsConclusions

Detail-on-demand video well-suited for Detail-on-demand video well-suited for training and home videotraining and home video Simple interaction style appropriate for Simple interaction style appropriate for

DVD-player interfacesDVD-player interfaces Enables wide range of authors due to emphasis on Enables wide range of authors due to emphasis on

ease of learning and use over richness of interactionease of learning and use over richness of interaction Early user study results of both authors and viewers Early user study results of both authors and viewers

very encouragingvery encouraging Hypervideo summaries Hypervideo summaries

Remove need for single context-free video summaryRemove need for single context-free video summary Multiple clip selection and link generation algorithmsMultiple clip selection and link generation algorithms Act as starting point for human-authored summaryAct as starting point for human-authored summary