Perceptive Context for Pervasive Computing Trevor Darrell Vision...
Transcript of Perceptive Context for Pervasive Computing Trevor Darrell Vision...
![Page 1: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/1.jpg)
Perceptive Context for Pervasive Computing
Trevor DarrellVision Interface GroupMIT AI Lab
![Page 2: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/2.jpg)
Perceptually Aware Displays
Camera associated with displayDisplay should respond to user
- font size- attentional load- passive acknowledgement
e.g., “Magic Mirror”, IntervalCompaq’s Smart KioskALIVE, MIT Media Lab
Camera
Display
![Page 3: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/3.jpg)
Example: A Face Responsive Display• Faces are natural interfaces!
- Ubiquitous, fast, expressive, general.- Want machines to generate and perceive faces.
• A Face Responsive Display...- Knows when it’s being observed- Recognizes returning observers- Tracks head pose- Robust to changing lighting, moving backgrounds…
![Page 4: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/4.jpg)
A Face Responsive Display
Tasks- Detection- Identification- Tracking
How? Exploit multiple visual modalities:- Shape- Color- Pattern
![Page 5: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/5.jpg)
Tasks and Visual Modalities
shape color pattern
detection silhouette classifier skin classifier face detection
identification biometrics flesh hue face recognition
tracking coarse motion estimation
clothing histogram
fine motion estimation / pose tracking
![Page 6: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/6.jpg)
Mode and Task Matrix
shape color pattern
detection silhouette classifier skin classifier face detection
identification biometrics flesh hue face recognition
tracking Shape change clothing histogram
Appearance change
![Page 7: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/7.jpg)
Finding Features
2D Head / hands localization- contour analysis: mark extremal points (highest curvature or
distance from center of body) as hand features- use skin color model when region of hand or face is found (color
model is independent of flesh tone intensity)
![Page 8: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/8.jpg)
Flesh color tracking
• Often the simplest, fastest face detector!• Initialize region of hue space
[ Crowley, Coutaz, Berard, INRIA ]
![Page 9: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/9.jpg)
Color Processing
• Train two-class classifier with examples of skin and not skin
• Typical approaches: Gaussian, Neural Net, Nearest Neighbor
• Use features invariant to intensityLog color-opponent [Fleck et al.]
(log(r) - log(g), log(b) - log((r+g)/2) )Hue & Saturation
![Page 10: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/10.jpg)
Flesh color tracking
Can use Intel OpenCV lib’s CAMSHIFT algorithm for robust real-time tracking.
(open source impl. avail.!)
[ Bradsky, Intel ]
![Page 11: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/11.jpg)
Intel’s computer vision library
![Page 12: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/12.jpg)
Detection with multiple visual modes
Find head sized peaksin 2-D or 3-D.
Detect skin pigment in hue-based color space
Classify intensity vector corresponding to face class
Shape
Flesh ColorDetection
Face PatternDetection
![Page 13: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/13.jpg)
Common Detection Failure Modes
Fooled by head shaped peaks
Fooled by flesh colored objects
Misses out of plane rotation or expression
Shape
Flesh ColorDetection
Face PatternDetection
![Page 14: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/14.jpg)
Robust real-time performance
Integrated Face Detection Algorithm(temporally asynch.
voting scheme)
Shape
Flesh ColorDetection
Face PatternDetection
![Page 15: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/15.jpg)
Mode and Task Matrix
shape color pattern
detection silhouette classifier skin classifier face detection
identification biometrics flesh hue face recognition
tracking Shape change clothing histogram
Appearance change
![Page 16: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/16.jpg)
A Key Technology: Video-Rate Stereo
• Two cameras −> stereo range estimation; disparity proportional to depth
• Depth makes tracking people easy- segmentation- shape characterization - pose tracking
• Real-time implementations becoming commercially available.
![Page 17: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/17.jpg)
Video-rate stereo
Foreground pixels; grouped by local connectivity
Computed disparity
Left and right images
![Page 18: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/18.jpg)
RGBZ input
![Page 19: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/19.jpg)
RGBZ input
![Page 20: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/20.jpg)
RGBZ input
![Page 21: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/21.jpg)
Range feature for ID!
• Body shape characteristics -- e.g., height measure.• Normalize for motion/pose: median filter over time
• Near future: full vision-based kinematic estimation and tracking--active research topic in many labs.
Trevor
MikeGaile
![Page 22: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/22.jpg)
Color feature for ID! For long-term tracking / identification, measure color hue and saturation
values of hair and skin….
For same-day ID, use histogram of entire body / clothing
Gaile Mike Trevor
![Page 23: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/23.jpg)
Mode and Task Matrix
shape color pattern
detection silhouette classifier skin classifier face detection
identification biometrics flesh hue face recognition
tracking Shape change clothing histogram
Appearance change
See lectures by Trevor later in the course
![Page 24: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/24.jpg)
Robust, Multi-modal Algorithm
Combine modules for detection:• Silhouette finds body• Color tracks extremities• Pattern discriminates head from hands.
Use each also to recognize returning people:• Face recognition • Biometrics (skeletal structure)• Hair and Skin hue• Clothing (intra-day.)
[ CVPR ‘98; T. Darrell, G. Gordon, M. Harville, J. Woodfill ]
![Page 25: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/25.jpg)
System Overview
![Page 26: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/26.jpg)
Classic Background Subtraction model
• Background is assumed to be mostly static• Each pixel is modeled as by a gaussian distribution in
YUV space• Model mean is usually updated using a recursive low-
pass filter
Given new image, generate silhouetteby marking those pixels that are significantlydifferent from the “background” value.
![Page 27: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/27.jpg)
Static Background Modeling Examples
[MIT Media Lab Pfinder / ALIVE System]
![Page 28: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/28.jpg)
Static Background Modeling Examples
[MIT Media Lab Pfinder / ALIVE System]
![Page 29: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/29.jpg)
Static Background Modeling Examples
[MIT Media Lab Pfinder / ALIVE System]
![Page 30: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/30.jpg)
The ALIVE System
UserVideoScreen
Autonomous Agents
Camera
![Page 31: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/31.jpg)
ALIVE • Real sensing for virtual world• Tightly coupled sensing-behavior-action• Vision routines: body/head/hand tracking
Kinematics /Rendering
Camera
Projector
Vision
Behaviors / Goals
User Agents
[ Blumberg, Darrell, Maes, Pentland, Wren, … 1995 ]
![Page 32: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/32.jpg)
ALIVE system, MIT
http://vismod.www.media.mit.edu/cgi-bin/tr_pagemaker (TR 257)
![Page 33: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/33.jpg)
http://vismod.www.media.mit.edu/cgi-bin/tr_pagemaker (TR 257)
![Page 34: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/34.jpg)
A Face Responsive Display
Video Display
StereoCameras
![Page 35: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/35.jpg)
Vision-only Application:Interactive Video Effects
![Page 36: Perceptive Context for Pervasive Computing Trevor Darrell Vision …courses.csail.mit.edu/6.869/lectnotes/lect20/lect20t... · 2005. 4. 21. · Perceptive Context for Pervasive Computing](https://reader035.fdocuments.net/reader035/viewer/2022071108/5fe38904394a34579b5c9ef9/html5/thumbnails/36.jpg)
end