Animal Tracking for Behavior Modeling
description
Transcript of Animal Tracking for Behavior Modeling
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
h
Animal Tracking for Behavior Modeling
James M. RehgGeorgia Tech
(standing in for Tucker Balch)
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hPeople
• PI Tucker Balch
• Faculty Jim Rehg, Computer Science
− Collaborator on animal tracking Aaron Bobick, Computer Science
− Collaborator on animal behavior modeling Bruce Walker, Psychology
− Collaborator on aquarium project Atsushi Nakazawa, Computer Science (Osaka Univ.,
Japan)− Collaborator on animal tracking
• PhD Students Michael Novitzky Jin Lee Matthew Flagg
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hGoals
•Track social animals reliably under natural conditions (in vivo) from video
•Estimate behavioral parameters from tracking data
•Construct executable models of social animal behavior
•Develop a biologically-inspired protocol for dynamic team formation
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hYear 1 Overview
•Goal Develop reliable multi-target tracking
algorithms for animals in video•Approach
Jointly estimate segmentation and motion of a nonrigid, deformable target
•Key requirements Reliably estimate target shape over time, to
support behavioral analysis Minimize the amount of human effort required
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hChallenges
•Accurate segmentation of target Set-point tracking is not sufficient
•Track reliably with significant camera motion Background subtraction is not sufficient
•Handle a wide range of animals Scalable solution for the animal kingdom
•Track multiple interacting targets Multiple instances of target type (e.g. wolf pack) Occlusions (with self, other targets, and
background)•Track long sequences, track across cuts
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hApproach
•Basic research in video object segmentation and tracking
•Modular software architecture Easily change features, models, and solver
•Testing in two scenarios Aquarium Monitoring
−Long sequences, static camera, some modeling effort Tracking Animal Planet videos
−Shorter sequences, moving camera, minimal human effort
•Preliminary application Accessible Aquarium Project
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hTracking App
Basic Trackers: -Contour feature w/ Iterated Closest Point-Color feature w/ Mean Shift
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hColor Histogram Tracker
• Fish Model: Appearance model For each species of fish,
multiple HSV Color Histograms on Image patches
Off-line Model Selection by Human
• Detection and Tracking: Histogram-based mean shift
approach Maximization of Bhattacharyya
Coefficient between Observation and Model
Selection of a model with the highest coefficient and update track
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hHistogram-based Model
Selection
Model Image Patches(to build histogram)Input Image Similarity
between shifted &
model patches
0.60.50.70.80.3 Best Model
*
Image Patch after Mean-shifting
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hAccessible Aquarium
Project•Provide a meaningful and informative aquarium
experience for visually-impaired or blind visitors•Approach
Track the movement of individual fish within tank Sonify the fish movement
•Example: 65 gallon marine aquarium Track yellow tangs and blue chromis Music structure is Bach chorale Each fish type is same instrument, different registers Movement speed mapped to note density (tempo) Horizontal dimension is stereo, vertical is timbre
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hExample
•Play video
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hTracking in the Video
Volume
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hGraphcut Tracking
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hTracking Results
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hSocial Game Retrieval
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hOverview
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hSummary of Progress
•Robust long-life tracking of multiple targets (fish) Tracking under controlled (but realistic)
conditions•Virtual Aquarium Project
Novel experience of animal behavior via sonification
•New state-of-art motion segmentation algorithm Accurate segmentation of (shorter) video
sequences under a wide range conditions
Robo
tics a
nd In
tellig
ent M
achi
nes @
Geo
rgia
Tec
hYear 2 Plans
•Make general tracking method work for longer sequences
•Develop integrated tracking application tool for biologists
•Connect to biologists more effectively (what to say about this?)
•Start developing behavior models, how?