Big worldofsmallmotions iccp13
Transcript of Big worldofsmallmotions iccp13
Imperceptible Changes and Motions in the World
Blood flow
Building and structuremovements
Camera moves due to motion of shutter and mirror
BreathingEye movements (Microsaccades)Michael Rubinstein, MIT 2013
Imperceptible Changes and Motions in the World
Low frequency motions Mid-range frequency motionsMichael Rubinstein, MIT 2013
Talk Overview
• Recap: Eulerian Video Magnification (SIGGRAPH’12)– With Hao-Yu Wu, Eugene Shih, John Guttag,
Fredo Durand, William T. Freeman
• EVM in the Wild
• Phase-based Video Motion Processing (SIGGRAPH’13)– With Neal Wadhwa, Fredo Durand, William T. Freeman
Michael Rubinstein, MIT 2013
Talk Overview
• Recap: Eulerian Video Magnification (SIGGRAPH’12)– With Hao-Yu Wu, Eugene Shih, John Guttag,
Fredo Durand, William T. Freeman
• EVM in the Wild
• Phase-based Video Motion Processing (SIGGRAPH’13)– With Neal Wadhwa, Fredo Durand, William T. Freeman
Michael Rubinstein, MIT 2013
Eulerian Video Processing
• Each pixel is processed independently
• We treat each pixel as a time series and apply signal processing to it
y
xtime
Michael Rubinstein, MIT 2013
Method Pipeline
Spat
ial
Dec
om
posi
tion
Spat
ial
Dec
om
posi
tion
Tem
pora
l fi
lter
ing
Spat
ial
Dec
om
posi
tion
Tem
pora
l fi
lter
ing
α1
α2
αn-1
αn
Σ
Σ
Σ
Σ
Spat
ial
Dec
om
posi
tion
Tem
pora
lfi
lter
ing
α1
α2
αn-1
Rec
onst
ruct
ion
αn
Σ
Σ
Σ
Σ
Michael Rubinstein, MIT 2013
Color Amplification Results
Source Color-amplified (x100)0.83-1 Hz (50-60 bpm)
Michael Rubinstein, MIT 2013
Bruce Wayne’s Pulse
Batman Begins (2005), courtesy of Warner Bros. Pictures
Michael Rubinstein, MIT 2013
Related Work: Pulse Detection in Videos
“Cardiocam” [Pho, Picard, McDuff 2010]
“Vital Signs Camera” – Philips(proprietary)
Michael Rubinstein, MIT 2013
Relating Temporal and Spatial Changes
Motion-magnified
Courtesy of Lili Sun
Michael Rubinstein, MIT 2013
Relating Temporal and Spatial Changes
Space
Inte
nsi
ty
0
Signal Shifted signal Motion-magnified signal
Michael Rubinstein, MIT 2013
Selective Motion Magnification in Natural Videos
Source(600 fps)
72-92 HzAmplified
Low E (82.4 Hz)
A (110 Hz)100-120 HzAmplified
Michael Rubinstein, MIT 2013
Related Work: Motion Magnification [Liu 2005]
Liu et al. Motion Magnification, SIGGRAPH 2005
Source Motion-magnified
Michael Rubinstein, MIT 2013
Related Work: Motion Magnification [Liu 2005]
+ +
++ +
Liu et al. Motion Magnification, 2005Michael Rubinstein, MIT 2013
Talk Overview
• Recap: Eulerian Video Magnification (SIGGRAPH’12)– With Hao-Yu Wu, Eugene Shih, John Guttag,
Fredo Durand, William T. Freeman
• EVM in the Wild
• Phase-based Video Motion Processing (SIGGRAPH’13)– With Neal Wadhwa, Fredo Durand, William T. Freeman
Michael Rubinstein, MIT 2013
EVM in the Wild: Pregnancy
Original Processed
“Tomez85” https://www.youtube.com/watch?v=J1wvFmWv7zYMichael Rubinstein, MIT 2013
EVM in the Wild: Pregnancy
“Tomez85” https://www.youtube.com/watch?v=gDpNN4g1klUMichael Rubinstein, MIT 2013
EVM in the Wild: Blood flow Visualization
Institute for Biomedical Engineering, Dresden Germanyhttps://www.youtube.com/watch?v=Nb18CRVmXGY
Red = high blood volumeBlue = low blood volume
Michael Rubinstein, MIT 2013
EVM in the Wild: Guinea Pig!
“SuperCreaturefan”: “Guinea pig Tiffany is the first rodent on Earth to undergo Eulerian Video Magnification.”http://www.youtube.com/watch?v=uXOSJvNwtIk
Source Motion-magnified
Michael Rubinstein, MIT 2013
Independent (Real-time) Ports
“webcam-pulse-detector”(Python + openCV)
+tracking
“VAmp - Video Amplifier”(Java)
Michael Rubinstein, MIT 2013
Talk Overview
• Recap: Eulerian Video Magnification (SIGGRAPH’12)– With Hao-Yu Wu, Eugene Shih, John Guttag,
Fredo Durand, William T. Freeman
• EVM in the Wild
• Phase-based Video Motion Processing (SIGGRAPH’13)– With Neal Wadhwa, Fredo Durand, William T. Freeman
Michael Rubinstein, MIT 2013
Phase-based Motion Magnification
Source LinearSIGGRAPH’12
Phase-basedSIGGRAPH’13
Michael Rubinstein, MIT 2013
Linear Pipeline (SIGGRAPH’12)
Laplacian pyramid[Burt and Adelson 1983]
Temporal filtering on intensities
Michael Rubinstein, MIT 2013
Phase-based Pipeline (SIGGRAPH’13)
Complex steerable pyramid[Portilla and Simoncelli 2000]
Temporal filtering on phases
PhaseAmplitude
Michael Rubinstein, MIT 2013
Improvement #1: More Amplification
Improves the bound by a factor of 4!
(derivation in the paper)
Amplification factor Motion in the sequence
Range of linear method:
Range of Phase-based method:
Michael Rubinstein, MIT 2013
Improvement #2: Better Noise Performance
Noise amplified Noise translated
Michael Rubinstein, MIT 2013
Results: Phase-based vs. Linear
Linear (SIGGRAPH’12) Phase-based (SIGGRAPH’13)
Clipping artifacts nearSharp edges and larger motions
Michael Rubinstein, MIT 2013
Results: Phase-based vs. Linear
Linear (SIGGRAPH’12) Phase-based (SIGGRAPH’13)Michael Rubinstein, MIT 2013
Phase-based Motion Attenuation
Source Linear Motion attenuation +Color amplification
Amplifies colorAnd motion jointly
Amplifies colorWithout amplifyingmotionMichael Rubinstein, MIT 2013
Phase-based Motion Attenuation
Source Phase-based motion attenuation
Courtesy of YouTube user ComputerPhysicsLabSimilar to Motion Denoising
[Rubinstein et al. 2010][Bai et al. 2012]
Michael Rubinstein, MIT 2013
Revealing Invisible Changes in the World
• NSF International Science and Engineering Visualization Challenge (SciVis), 2012
• Science Vol. 339 No. 6119 Feb 1 2013
Michael Rubinstein, MIT 2013
Conclusions
• The world is full of small motions and changes we cannot normally see
• We develop algorithms to analyze and visualize them through videos
– Many potential uses in medical applications and scientific analysis
• Phase-based Video Motion Processing (to be presented at SIGGRAPH’13)
– More magnification, less noise
– Phase-based motion analysis is around for a while (Fleet and Jepson 1990) but not commonly used for editing
• New paper and code available soon!
Michael Rubinstein, MIT 2013