Activity Monitoring October 19-20, 1999 DARPADARPA Bob Bolles, Brian Burns, Marty Fischler, Ravi...
-
Upload
rodger-beasley -
Category
Documents
-
view
223 -
download
0
Transcript of Activity Monitoring October 19-20, 1999 DARPADARPA Bob Bolles, Brian Burns, Marty Fischler, Ravi...
Activity Monitoring
October 19-20, 1999
DARPADARPADARPADARPA
Bob Bolles, Brian Burns,Marty Fischler, Ravi Gopalan,
Marsha Jo Hannah, Dave ScottSRI International
Rama Chellappa, Yiannis Aloimonos, Doug Ayers,
Ross Cutler, Larry Davis,Azriel Rosenfeld, Chandra Shekhar
University of Maryland
2October 19-20, 1999
Application Challenge
Develop techniques for dramatically increasing the productivity of an aerial video analyst.
3October 19-20, 1999
High-Level Approach
Sensor Multiplexing to Monitor Several Sites “Simultaneously” and Semi-automatically
4October 19-20, 1999
Technical Goalfor Activity Monitoring
Develop techniques to monitor sites, such as cantonment areas and tree lines, from an airborne platform and identify tactically significant activities involving people and vehicles.
Sample Activities:• people entering a forbidden area• people congregating near an embassy• vehicles convoying along a road• people readying a missile for launch
5October 19-20, 1999
Technical Challengesfor Activity Monitoring
• Representation of activities
• Recognition of activities from a moving platform
• Moving object classification
Activity
A large tactical vehicle exiting a hide site (along a tree line). People are often visible guiding the vehicle out.
Starting search
Looking for people
Detect person(s)
Looking for large vehicle
All people leave the FOV
Exit of large vehicle detected
Detect small vehicle
Activity Template
Zoom to a NFOV &aim close to tree line
Move to new pointalong tree line
Detect large vehicle
6October 19-20, 1999
Approach
Task specification•Retrieve or sketch a site model (roads, buildings,…)•Specify the task (what, where, when, & reports/alarms)
Automatic monitoring•Scan the appropriate area •Stabilize the video (MTS -- Sarnoff)•Register the video to the site model (PVR -- Harris)•Detect and track moving objects•Characterize & classify the tracked objects•Recognize activities•Report tactically significant events
AMIS -- Activity Monitoring Integrated Systesm
7October 19-20, 1999
Site Model
Site model
Task specification
Scan area
Stabilize video
Register video
Track objects
Characterize objects
Recognize activities
Report events
Powers Road
Mosby Road
Motorpool
Berm
“Residence” Area
8October 19-20, 1999
Task Specification
Site model
Task specification
Scan area
Stabilize video
Register video
Track objects
Characterize objects
Recognize activities
Report events
Drivers jog to their vehicles
Vehicles drive away
Drivers jog to their vehicles
Motorpool
Residence Area
9October 19-20, 1999
Scan Area
Site model
Task specification
Scan area
Stabilize video
Register video
Track objects
Characterize objects
Recognize activities
Report events
Motorpool
Residence Area
Sensor Field of View
10October 19-20, 1999
Stabilize Video
Site model
Task specification
Scan area
Stabilize video
Register video
Track objects
Characterize objects
Recognize activities
Report events
Raw Video
11October 19-20, 1999
Stabilize Video
Site model
Task specification
Scan area
Stabilize video
Register video
Track objects
Characterize objects
Recognize activities
Report events
Stabilized Video
12October 19-20, 1999
Register Video
Site model
Task specification
Scan area
Stabilize video
Register video
Track objects
Characterize objects
Recognize activities
Report events
Desired field of view
Actual field of view
13October 19-20, 1999
Track Objects
Site model
Task specification
Scan area
Stabilize video
Register video
Track objects
Characterize objects
Recognize activities
Report events
14October 19-20, 1999
Characterize Objects
Site model
Task specification
Scan area
Stabilize video
Register video
Track objects
Characterize objects
Recognize activities
Report events
Object Properties
•Size, velocity, …
•Articulation -- periodicity
(for animate/inanimate)
•Could it be parallax?
•Color, shape, …
•Location in the site
15October 19-20, 1999
Report Events
Site model
Task specification
Scan area
Stabilize video
Register video
Track objects
Characterize objects
Recognize activities
Report events
People moving down Powers Road
Vehicles leavingmotorpool area
People approachingmotorpool area
People enteredmotorpool areaAlert: Battle Group Pullout
16October 19-20, 1999
Primary Contributions
• Representation and recognition of activities (in the context of a site model)
– augmented finite state machines
– dynamic belief networks
• Moving object classification components– parallax analysis
– animate/inanimate classification
– velocity, size, ...
17October 19-20, 1999
Introduction toLive Flight Experiments
18October 19-20, 1999
Activity Monitoring
1. Battle group pullout
2. Battle group return
3. People exiting woods near berm
4. People crossing the road
Berm
“Residence” Area
Activities
Motorpool
19October 19-20, 1999
Activity Templates
Event Primitives– Approaching/Leaving– Gaining-Ground/
Losing-Ground– Entering/Exiting– Moving-inside-region– Temporal durations
Combinations– Boolean operations– Sequences– Graphs
Starting search
Looking for people
Detect person(s)
Looking for large vehicle
All people leave the FOV
Exit of large vehicle detected
Detect small vehicle
Activity Template
Zoom to a NFOV &aim close to tree line
Move to new pointalong tree line
Detect large vehicle
20October 19-20, 1999
Site Model Sketching
21October 19-20, 1999
Video Registration
Image
World
22October 19-20, 1999
Activity Analysisin World Coordinates
Image
World
23October 19-20, 1999
Moving Object Detection
Raw video fields
Raw differences
AND’d differences
Image N
24October 19-20, 1999
Parallax Versus Independent Motion
25October 19-20, 1999
Animate/Inanimate
Periodicity analysis
26October 19-20, 1999
Align and scale objects
Compute similarity matrix S
Template fit peaks of S
Track objects
Autocorrelate S
T r ead m il l
1 0 20 3 0 40 5 0 60 7 0 80 9 0 1 00
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
1 00
Tre admill
10 20 30 40 5 0 60
10
20
30
40
50
60
Periodicity Analysisfor Classifying Objects as
Animate or Inanimate
27October 19-20, 1999
Parallax Detection
Flagged as being locally consistent with “motion parallax”
28October 19-20, 1999
AM’s Windows
29October 19-20, 1999
Stabilization Params
Metadata
MTS-Ground
Multiple Target Surveillance
Precision VideoRegistration
Raw Video (analog)
CAGS-Ground
CAGS-Air
Ground Station
Activity Monitoring
Air-Ground Partitionfor 1999
30October 19-20, 1999
Battle Group Pullout
1. Battle group pullout
2. Battle group return
3. People exiting woods near berm
4. People crossing the road
Activities
Drivers jog to their vehiclesDrivers jog to their vehicles
Vehicles drive away
31October 19-20, 1999
Battle Group Return
Vehicles return & park
Drivers walk back to residence
1. Battle group pullout
2. Battle group return
3. People exiting woods near berm
4. People crossing the road
Activities
32October 19-20, 1999
People Exiting Woods near Berm
People Exit Tree Line
1. Battle group pullout
2. Battle group return
3. People exiting woods near berm
4. People crossing the road
Activities
33October 19-20, 1999
People Crossing Road
People Exit Tree Line and cross the road
1. Battle group pullout
2. Battle group return
3. People exiting woods near berm
4. People crossing the road
Activities
34October 19-20, 1999
PreliminaryEvent Statistics
E vent Trials Successes Success Rate
Approaching 8 8 100%
Entering 14 13 93%
Moving Inside 16 16 100%
Exiting 13 10 77%
Leaving 8 8 100%
Totals 59 55 93%
• Results from 2 flights with high contrast imagery
35October 19-20, 1999
PreliminaryWhole Vignette Statistics
Vignette Trials Successes Success Rate
Battle Group Pullout 5 5 100%
Battle Group Return 5 4 80%
People Exit Woods 1 1 100%
People Cross Road 6 3 50%
People Stealing Vehicles 3 3 100%
Totals 20 16 80%
• Results from 2 flights with high contrast imagery
36October 19-20, 1999
Summary
Accomplishments:• AMIS – Activity Monitoring Integrated System• Activity Templates – an initial representation for activities• An initial technique for recognizing activities based on augmented
finite state machines• An extension to dynamic belief networks to activity recognition• A technique for identifying moving objects due to motion parallax • A technique for classifying moving objects as animate or inanimate• A semi-automatic video registration technique• A realtime moving object detection technique
Increase the productivity of an image analyst by a factor of 10 to 15 by multiplexing a high-performance sensor and automatically identifying potentially significant activities.
Goal:
37October 19-20, 1999
Evaluation of‘99 Accomplishments
Moving object classification -- Components only
Sensor Control -- manual versus computer-controlled
HCI -- primarily on PC, not integrated into CAGS-Ground
38October 19-20, 1999
Plans for ‘00
• Represent & recognize more complex activities, such as checkpoint monitoring
• Call PVR for video registration
• Place sensor under computer-control (based on MTS results)
• Integrate moving object classification
• Integrate the HCI into CAGS-Ground