Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer...
-
Upload
shon-mcgee -
Category
Documents
-
view
215 -
download
0
Transcript of Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer...
![Page 1: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/1.jpg)
Foreground ModelingThe Shape of Things that
Came
Nathan JacobsAdvisor: Robert Pless
Computer ScienceWashington University in St. Louis
![Page 2: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/2.jpg)
Nathan Jacobs 2
Visual Surveillance
• Observe people and vehicles– Where are they?– Where have they been?– Where are they going?
• Answering these questions requires object tracking.
![Page 3: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/3.jpg)
Nathan Jacobs 3
Probabilistic Tracking
• Tracking is commonly cast in a Bayesian framework to estimate object shape and location– Initial estimate = combination of image data likelihood and
initialization prior– Updated estimate = combination of image data likelihood and
state prediction prior
• Likelihood functions are the focus of most tracking work– Color histograms, templates
• Our focus is on the prior terms
![Page 4: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/4.jpg)
Nathan Jacobs 4
Quotes from yesterday
• “Initialization of tracking is important but not addressed here.”
• “Our object model assumes a well calibrated camera and a flat-ground plane.”
• “The prior term is a tricky thing to design.”
![Page 5: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/5.jpg)
Nathan Jacobs 5
Passive Vision : The Big Picture
• Learn strong scene-specific priors by watching the same scene for a long time– Made easier because the cameras are static– Should be learned online
• Priors can be used to improve anomaly detection and tracking algorithms
![Page 6: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/6.jpg)
Nathan Jacobs 6
Scene-specific Motion Priors
![Page 7: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/7.jpg)
Nathan Jacobs 7
Unusual Traffic Motion
Video segment with anomalous motion (an ambulance using the median to pass stopped cars).
False color sequence highlighting anomalous motions.
![Page 8: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/8.jpg)
Nathan Jacobs 8
Online Prior Learning for Tracking
Online learning and use of motion priors:
• reduces the number of particles needed
• increases the number of objects that can be tracked. Frames
Obj
ects
![Page 9: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/9.jpg)
Nathan Jacobs 9
What else can we model?
• Watching for a long time allows us to build models of– Pixel intensity– Image derivatives– Image motion patterns
• We now transition to features based on the shape of foreground objects
![Page 10: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/10.jpg)
Nathan Jacobs 10
An Example Video
![Page 11: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/11.jpg)
Nathan Jacobs 11
Generating Examples Shapes
Current FrameForeground Mask Shape Descriptor
Background Image
For a long time:
1. Detect foreground objects
2. Generate a shape descriptor for each object
3. Add shape descriptor to training set
![Page 12: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/12.jpg)
Nathan Jacobs 12
Shape Descriptor
• Currently using a simple shape model– A 20-dimensional feature vector– Each dimension is the distance from center to edge
of object
• Other shape models are possible
![Page 13: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/13.jpg)
Nathan Jacobs 13
Two Shape Model Types
• Both models are PCA subspaces
• Global model– Subspace is location independent – Distribution estimate is location dependent
• Local model– Subspace and distribution are both location
dependent
![Page 14: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/14.jpg)
Nathan Jacobs 14
A location-independent Shape Basis
Generate shape subspace using PCA on all shapes in training set.
First Principle Component
(~size)
Second Principle Component
(~orientation)
![Page 15: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/15.jpg)
Nathan Jacobs 15
Location-dependent Coefficients
First Principle Component
(~size)
Second Principle Component
(~orientation)
![Page 16: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/16.jpg)
Nathan Jacobs 16
Location-specific Shape Subspaces
Generate a shape subspace using shapes found in a small region of the image.
Location-specific mean shapes
![Page 17: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/17.jpg)
Nathan Jacobs 17
Location-specific Shape Subspaces
Shape subspaces are location dependent.
Much smaller variations in some regions.
First PC Variations Second PC Variations
![Page 18: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/18.jpg)
Nathan Jacobs 18
Shapes in (Shape) Space
![Page 19: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/19.jpg)
Nathan Jacobs 19
An Example from PETS
First Principle Component
(~size)
Second Principle Component(~aspect)
![Page 20: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/20.jpg)
Nathan Jacobs 20
Location-dependent Mean Shapes
Mean Shapes
![Page 21: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/21.jpg)
Nathan Jacobs 21
Location-dependent Subspaces
First PC Variations Second PC Variations
![Page 22: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/22.jpg)
Nathan Jacobs 22
Object Initialization for Tracking
• Object initialization is a crucial step of any tracking algorithm
• Use shape priors to determine object boundaries– Combines image information and shape prior– Penalize unlikely shapes– More accurate than image information alone
• Major point: strong priors make simple methods work
![Page 23: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/23.jpg)
Nathan Jacobs 23
Object Boundary Detection
• Goal is to determine object boundaries to improve tracking initialization
• Algorithm– Find candidates using background subtraction– Initialize each candidate with a location-
specific mean shape– Optimize shape by gradient descent in PCA
shape subspace (penalize object overlap)• Image data term: sum of per-pixel foreground
probability inside shape• Shape prior term: sum of absolute value of PCA
coefficients
![Page 24: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/24.jpg)
Nathan Jacobs 24
Segmentation Results
Subspace only Subspace and Prior
Global shape model
Local shape model
![Page 25: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/25.jpg)
Nathan Jacobs 25
Summary
1. Static cameras give strong priors.
2. Unsupervised training of a localized shape prior is possible.
3. Localized shape priors can be used to improve object initialization for tracking.
![Page 26: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/26.jpg)
Background
![Page 27: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/27.jpg)
Nathan Jacobs 27
Efficient Segmentation
• Use gradient descent in low-dimensional shape subspace
• Gradient estimation– For each underlying
shape parameter• Sum along two edges
of polygon– For PC components and
object location• Weighted combination
of polygon edge scores
![Page 28: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/28.jpg)
Nathan Jacobs 28
Choice of Support Region
![Page 29: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/29.jpg)
Nathan Jacobs 29
Choosing Constants for Updating Prior Models
Threshold
.99999
0.9999
0.999
0.99
0.9
The best learning rate depends on scene,
application, time-of-day, weather, image location.
Slow update Fast updateCurrent Frame
VSSN 2006
![Page 30: Foreground Modeling The Shape of Things that Came Nathan Jacobs Advisor: Robert Pless Computer Science Washington University in St. Louis.](https://reader030.fdocuments.net/reader030/viewer/2022032723/56649d005503460f949d2d11/html5/thumbnails/30.jpg)
Nathan Jacobs 30
Segmentation Energy Function
Minimize
Penalty on size
Per-pixel foreground likelihood
Shape penalty based on prior (sum of PCA coefficients)