Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs
-
Upload
marat-zhanikeev -
Category
Technology
-
view
466 -
download
0
description
Transcript of Methods and Algorithms for Automatic Slicing of Screen Capture Video using Slide PDFs
.
YALMS: Review
.YALMS......
....stands forYetAnotherLectureManagementSystem
• basically, a class robot◦ robot: set of automation scripts and frameworks
• first presented inET研2013.07 04
• a long-term project with 4-5 main components -- see the roadmap at the end
04 M.Zhanikeev+1 "YALMS: Yet Another Lecture Management System..." IEICE ET研 (2013.09)
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 2/28...
2/28
.
YALMS: Overall Design
Step 1: Record and Stream
VA
SLecturer
Viewers
ContentServer
HTML5
HTML5Muxer
RecordScreenStreaming
Step 2: Annotate
Lecturer
S
Auto-annotate
C
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 3/28...
3/28
.
YALMS: Annotation
VIDEO
V A S
Annotate
Pack/Encode Screen
VIDEO
V A S
V A Realtime Screen
VIDEO
S Mux
A V MISC. C
C
PNG
Replay Screen
VIDEO
A V MISC. C
Annotate
S
S
S The focus of this study YALMS Robot
S
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 4/28...
4/28
.
Fuzzy Environments
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 5/28...
5/28
.
Sources of Fuzziness
1. unusual screen sizes in presentation mode
◦ this PDF was created in16:9 mode to fit my notebook perfectly, will NOT fitothers
2. unpredictable or unavoidablemargins when saving your slides as PDF◦ PPTX! no way to avoid white margins in print◦ tex2pdf creates perfect PDFs! --- look the same in PDF and in presentation mode
3. slide animation◦ PPTX animation is really BAD! (confusing, difficult to print, ...)◦ latex/beamer creates sequences of pages for print -- same page number, many pages◦ the point is ... animation should look good both in print and on screen
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 6/28...
6/28
.
Objectives and Assumptions
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 7/28...
7/28
.
Objectives and Assumptions
.YALMS should be able.....
.... to retain automation capability in fuzzy environments
• ASSUMPTION 1: no rules for slide PDFs -- only the PDF formatitself
• ASSUMPTION 2: no rules for screen video -- black margins, wierdsizes -- all OK
• ASSUMPTION 3: no additional information is requested,
annotation should happen with only (1) and (2)
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 8/28...
8/28
.
Tools and Actual Software
1. pdftk : a nice library for handing PDFs 07
2. ImageMagic : the best graphics library (basically free Photoshop) 08
3. VLC for screen capture, ffmpeg will also work
4. ffmpeg for video editing 06
5. all scripting is done in php -- suitable for both CLI and web server
07 "pdftk: The PDF toolkit" http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit (2013)
08 "ImageMagic Homepage" http://www.imagemagick.org/script/index.php (2013)
06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013)
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 9/28...
9/28
.
Solutions
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 10/28...
10/28
.
Solution 1: Simple Scene Detection
• the simplest way to detect scene detection is to extract all I-Frames 06
• createsmore images than actual scene changes, but redundancy is not aproblem
$ ffmpeg -i screen.mp4 -vf select="eq(pict_type¥,I)" -vsync 0 -an frames.%03d.png
$ stands for command prompt, all commands are single-line
06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013)
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 11/28...
11/28
.
Solution 2: Trim the Black Margins
• deal with fuzzy margins and sizes using ImageMagic's
convert• OK to add a small border because trim does not cut everything anyway
• force frame resize
$ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 12/28...
12/28
.
Solution 2: Trim Performance
Nothing to trim
Trim success!
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 13/28...
13/28
.
Solution 3: Bring to Common Size• cannot compare without same size
$ convert -colorspace gray -bordercolor black -border 8x5 -resize 1280x720! -density 300x300 -quality 100 pages.002.pdf pages.002.png
$ stands for command prompt, all commands are single-line
$ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 14/28...
14/28
.
Solution 3: Size Automation
• easy to find page size in command line, so, automation OK
platypus@platypus-PC /local/yalms.content$ identify -format "%w %h" frames.004.png1280 720
platypus@platypus-PC /local/yalms.content$ identify -format "%w %h" frames.004c.png1156 720
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 15/28...
15/28
.
Solution 4: Compare using the AE Metric
• AE metric: simply the absolute pixel diff between two images
• alternatives: PNSR, MLSE ...
$ compare -colorspace gray -metric AE -fuzz 50% pages.002.png frames.004b.png diff.png
7021
$ stands for command prompt, all commands are single-line
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 16/28...
16/28
.
Solutions: Example
Video frameVideo frame(trim, resize)
PDF page (border, resize)
Diff imageAE: 7019
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 17/28...
17/28
.
Parameters and Performance
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 18/28...
18/28
.
Performance: Experimental Setup
• a 1.5h class
• screen capture and slide PDF
• fuzzy: 16:10 screen size (black margin)
• fuzzy: many ALT-TAB switches between applications
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 19/28...
19/28
.
Performance: Parameters and Metrics
• fuss: used by ffmpeg as comparison threshold
• pace : minimum frequency of page changes -- possible to remove jitter
• performance: annotation success
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 20/28...
20/28
.
Performance: Frame Diffs
• generally, good
trend, becauseonly one framematches well
• not ideal performance
0 20 40 60 80 100Ordered list of frame diffs
1.2
2.4
3.6
4.8
6
log(
fram
e d
iff)
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 21/28...
21/28
.
Performance: Results
• thickness:
fuss value(between 10% and 90%)
• changing fuss can helpsometimes
• some cases arepersistentfailures
0 2 4 6 8 10 12 14 16 18Page sequence
0
0.5
1
1.5
2
2.5
3
3.5
log(
tim
e d
iff)
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 22/28...
22/28
.
Performance: Room for Improvement?
• multi-fuss comparison, pick the best
• when failing consistently, just setmarker between two neighboringsuccesses◦ mark it as failed and ask for human
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 23/28...
23/28
.
YALMS.annotation: The Robot
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 24/28...
24/28
.
YALMS.annotation : The Robot
screen video
SlidesPDF
LaTeX/BeamerKeynote
PowerPoint…
VLCffmpeg
I-Frames I-Frames
pages PNGs
ANNOTATION
slices
PNGstimestamps
End productscenes cleanup
splitpdf2png
03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013)
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 25/28...
25/28
.
YALMS.annotation : The Script
$ php robot.phpPURPOSE: to process two files: SLIDES PDF and
SCREEN CAPTURE MOVIEOUTCOME: ANNOTATIONS for the movie -- times for each
PDF page in the movie[pace] minimum time gap between page changes
-- will ignore fickers below that threshold[wdir] full path to working directory[screen] filename of the screen capture movie
-- should be in $wdir[slides] filename of the slides PDF file -- should be in $wdir$
$ stands for command prompt, all commands are single-line
03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013)
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 26/28...
26/28
.
YALMS: The Roadmap
YALMS: Intro
YALMS.annotation
2013.07・ET研
2013.09・ET研
YALMS.streaming2014.01・ET研
YALMS.hiperf2013.10・ITS-MMS研
ET研
その他YALMS.cloud
2013.11・IN研
YALMS.format
YALMS.done!
DASH Adaptive HTML5
PDF slices
VM sync groupdrive
SSDmulticore
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 27/28...
27/28
.
That’s all, thank you ...
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 28/28...
28/28
.
[01] (2013)The enPiT Projecthttp://www.enpit.jp
[02] (2013)YALMS Project Homehttps://github.com/maratishe/yalms
[03] (2013)YALMS Annotation Robot (software behind this paper)https://github.com/maratishe/yalms.annotation
[04] M.Zhanikeev+1 (2013.09)YALMS: Yet Another Lecture Management System...IEICE ET研
[05] (2013)Homepage of edubase Stream at NIIhttp://stream.edubase.jp
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 28/28...
28/28
.
[06] (2013)Homepage of ffmpeg and ffserver toolshttp://www.ffmpeg.org
[07] (2013)pdftk: The PDF toolkithttp://www.pdflabs.com/tools/pdftk-the-pdf-toolkit
[08] (2013)ImageMagic Homepagehttp://www.imagemagick.org/script/index.php
M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 28/28...
28/28