1 Data Mining for Surveillance Applications Suspicious Event Detection Dr. Bhavani Thuraisingham...

1

Data Mining for Surveillance Applications

Suspicious Event Detection

Dr. Bhavani Thuraisingham

April 2006

2

OutlineOutline

Acknowledgements Acknowledgements Data Mining for Security Data Mining for Security

ApplicationsApplications Surveillance and Suspicious Event Surveillance and Suspicious Event

DetectionDetection Directions for SurveillanceDirections for Surveillance Other applicationsOther applications

3

AcknowledgementsAcknowledgements

Prof. Latifur KhanProf. Latifur Khan Gal LaveeGal Lavee Ryan LayfieldRyan Layfield Sai Chaitanya Sai Chaitanya

4Data Mining for Security Data Mining for Security ApplicationsApplications

Data Mining has many applications in Data Mining has many applications in Cyber Security and National SecurityCyber Security and National Security Intrusion detection, worm detection, Intrusion detection, worm detection,

firewall policy managementfirewall policy management Counter-terrorism applications and Counter-terrorism applications and

SurveillanceSurveillance Fraud detection, Insider threat analysisFraud detection, Insider threat analysis

Need to enforce security but at the Need to enforce security but at the same time ensure privacysame time ensure privacy

5Data Mining for Data Mining for

SurveillanceSurveillanceProblems AddressedProblems Addressed

Huge amounts of Huge amounts of surveillance and video surveillance and video data available in the data available in the security domainsecurity domain

Analysis is being done off-Analysis is being done off-line usually using “Human line usually using “Human Eyes”Eyes”

Need for tools to aid Need for tools to aid human analyst ( pointing human analyst ( pointing out areas in video where out areas in video where unusual activity occurs)unusual activity occurs)

6

ExampleExample

Using our proposed system:Using our proposed system:

Greatly Increase video analysis Greatly Increase video analysis efficiencyefficiency

User Defined

Event of interest

Video Data

Annotated Video w/ events of interest highlighted

7

The Semantic GapThe Semantic Gap

The disconnect between the low-level The disconnect between the low-level features a machine sees when a video is features a machine sees when a video is input into it and the high-level semantic input into it and the high-level semantic concepts (or events) a human being sees concepts (or events) a human being sees when looking at a video clip when looking at a video clip

Low-Level featuresLow-Level features: : color, texture, color, texture, shapeshape

High-level semantic conceptsHigh-level semantic concepts: : presentation, newscast, boxing matchpresentation, newscast, boxing match

8

Our ApproachOur Approach Event Representation Event Representation

Estimate distribution of pixel intensity Estimate distribution of pixel intensity change change

Event ComparisonEvent Comparison Contrast the event representation of Contrast the event representation of

different video sequences to determine if different video sequences to determine if they contain similar semantic event content.they contain similar semantic event content.

Event DetectionEvent Detection Using manually labeled training video Using manually labeled training video

sequences to classify unlabeled video sequences to classify unlabeled video sequences sequences

9

Event RepresentationEvent Representation Measures the quantity and type of changes occurring Measures the quantity and type of changes occurring

within a scene within a scene A video event is represented as a set of x, y and t intensity A video event is represented as a set of x, y and t intensity

gradient histograms over several temporal scales.gradient histograms over several temporal scales. Histograms are normalized and smoothedHistograms are normalized and smoothed

10

Event ComparisonEvent Comparison Determine if the two video sequences Determine if the two video sequences

contain similar high-level semantic contain similar high-level semantic concepts (events). concepts (events).

Produces a number that indicates how Produces a number that indicates how close the two compared events are to one close the two compared events are to one another. another.

The lower this number is the closer the two The lower this number is the closer the two events are. events are.

22 1 2

, , 1 2

[ ( ) ( )]1

3 ( ) ( )

l lk kl l

k l i k k

h i h iD

L h i h i

11

Event DetectionEvent Detection

A robust event detection system A robust event detection system should be able toshould be able to Recognize an event with reduced Recognize an event with reduced

sensitivity to actor (e.g. clothing or skin sensitivity to actor (e.g. clothing or skin tone) or background lighting variation.tone) or background lighting variation.

Segment an unlabeled video containing Segment an unlabeled video containing multiple events into event specific multiple events into event specific segmentssegments

12

Labeled Video EventsLabeled Video Events

These events are manually labeled These events are manually labeled and used to classify unknown eventsand used to classify unknown events

Walking1 Walking1 Running1Running1 Waving2Waving2

13

Labeled Video EventsLabeled Video Events

walkinwalkin

g1g1walkinwalkin

g2g2walkinwalkin

g3g3runninrunnin

g1g1runninrunnin

g2g2runninrunnin

g3g3runninrunnin

g4g4waving waving

22

walkinwalking1g1 00 0.276250.27625 0.245080.24508 1.22621.2262 1.3831.383 0.974720.97472 1.37911.3791 10.96110.961

walkinwalking2g2 0.276250.27625 00 0.178880.17888 1.47571.4757 1.50031.5003 1.29081.2908 1.5411.541 10.58110.581

walkinwalking3g3 0.245080.24508 0.178880.17888 00 1.12981.1298 1.09331.0933 0.886040.88604 1.12211.1221 10.23110.231

runninrunning1g1 1.22621.2262 1.47571.4757 1.12981.1298 00 0.438290.43829 0.304510.30451 0.398230.39823 14.46914.469

runninrunning2g2 1.3831.383 1.50031.5003 1.09331.0933 0.438290.43829 00 0.238040.23804 0.107610.10761 15.0515.05

runninrunning3g3 0.974720.97472 1.29081.2908 0.886040.88604 0.304510.30451 0.238040.23804 00 0.204890.20489 14.214.2

runninrunning4g4 1.37911.3791 1.5411.541 1.12211.1221 0.398230.39823 0.107610.10761 0.204890.20489 00 15.60715.607

wavingwaving22 10.96110.961 10.58110.581 10.23110.231 14.46914.469 15.0515.05 14.214.2 15.60715.607 00

14

Experiment #1Experiment #1

Problem: Recognize and classify events Problem: Recognize and classify events irrespective of direction (right-to-left, left-irrespective of direction (right-to-left, left-to-right) and with reduced sensitivity to to-right) and with reduced sensitivity to spatial variations (Clothing)spatial variations (Clothing)

““Disguised Events”- Events similar to Disguised Events”- Events similar to testing data except subject is dressed testing data except subject is dressed differentlydifferently

Compare Classification to “Truth” Compare Classification to “Truth” (Manual Labeling)(Manual Labeling)

15


Classification: WalkingClassification: Walking

Disguised Walking 1

walking1walking1 walking2walking2 walking3walking3 running1running1 running2running2 running3running3 running4running4 waving2waving2

0.976530.97653 0.451540.45154 0.596080.59608 1.54761.5476 1.46331.4633 1.57241.5724 1.54061.5406 12.22512.225

16


Classification: RunningClassification: Running

Disguised Running 1


1.4111.411 1.38411.3841 1.06371.0637 0.567240.56724 0.974170.97417 0.935870.93587 1.09571.0957 11.62911.629

17Classifying Disguised Classifying Disguised EventsEvents

Classification: RunningClassification: Running

Disguised Running 3


1.30491.3049 1.00211.0021 0.880920.88092 0.81140.8114 1.10421.1042 1.11891.1189 1.09021.0902 12.80112.801


Classification: WavingClassification: Waving

Disguised Waving 1


13.64613.646 13.11313.113 13.45213.452 18.61518.615 19.59219.592 18.62118.621 20.23920.239 2.24512.2451


DisguiseDisguisewalking1walking1

DisguiseDisguisewalking2walking2

DisguiseDisguiserunning1running1



DisguiseDisguisewaving1waving1

DisguiseDisguisewaving2waving2

DisguiseDisguisewalking1walking1 00 0.193390.19339 1.21591.2159 0.859380.85938 0.675770.67577 14.47114.471 13.42913.429

DisguiseDisguisewalking2walking2 0.193390.19339 00 1.43171.4317 1.18241.1824 0.955820.95582 12.29512.295 11.2911.29

DisguiseDisguiserunning1running1 1.21591.2159 1.43171.4317 00 0.375920.37592 0.451870.45187 15.26615.266 15.00715.007

DisguiseDisguiseRunning2Running2 0.859380.85938 1.18241.1824 0.375920.37592 00 0.133460.13346 16.7616.76 16.24716.247

DisguiseDisguiseRunning3Running3 0.675770.67577 0.955820.95582 0.451870.45187 0.133460.13346 00 16.25216.252 15.62115.621

DisguiseDisguisewaving1waving1 14.47114.471 12.29512.295 15.26615.266 16.7616.76 16.25216.252 00 0.458160.45816

DisguiseDisguisewaving2waving2 13.42913.429 11.2911.29 15.00715.007 16.24716.247 15.62115.621 0.458160.45816 00

20


This method yielded 100% Precision This method yielded 100% Precision (i.e. all disguised events were (i.e. all disguised events were classified correctly). classified correctly).

Not necessarily representative of the Not necessarily representative of the general event detection problem.general event detection problem.

Future evaluation with more event Future evaluation with more event types, more varied data and a larger types, more varied data and a larger set of training and testing data is set of training and testing data is needed needed

21


Problem: Given an unlabeled video sequence describe the high-Problem: Given an unlabeled video sequence describe the high-level events within the videolevel events within the video

Capture events using a sliding window of a fixed width (25 frames Capture events using a sliding window of a fixed width (25 frames in example)in example)

22


Running Similarity GraphRunning Similarity Graph

23


Walking Similarity GraphWalking Similarity Graph

24Recognizing Events in Recognizing Events in Unknown Video SegmentUnknown Video Segment

Waving Similarity Waving Similarity GraphGraph

25


Minimum Similarity Graph

Walking Running Waving Running

26

XML Video AnnotationXML Video Annotation Using the event detection scheme we generate a Using the event detection scheme we generate a

video description document detailing the event video description document detailing the event composition of a specific video sequencecomposition of a specific video sequence

This XML document annotation may be replaced This XML document annotation may be replaced by a more robust computer-understandable by a more robust computer-understandable format (e.g. the VEML video event ontology format (e.g. the VEML video event ontology language). language).

<?xml version="1.0" encoding="UTF-8"?><?xml version="1.0" encoding="UTF-8"?><videoclip><videoclip> <Filename>H:\Research\MainEvent\<Filename>H:\Research\MainEvent\ Movies\test_runningandwaving.AVI</Filename>Movies\test_runningandwaving.AVI</Filename> <Length>600</Length><Length>600</Length> <Event><Event> <Name>unknown</Name><Name>unknown</Name> <Start>1</Start><Start>1</Start> <Duration>106</Duration><Duration>106</Duration> </Event></Event> <Event><Event> <Name>walking</Name><Name>walking</Name> <Start>107</Start><Start>107</Start> <Duration>6</Duration><Duration>6</Duration> </Event></Event></videoclip></videoclip>

27

Video Analysis ToolVideo Analysis Tool Takes annotation document as input and organizes the Takes annotation document as input and organizes the

corresponding video segment accordingly.corresponding video segment accordingly. Functions as an aid to a surveillance analyst searching for Functions as an aid to a surveillance analyst searching for

“Suspicious” events within a stream of video data.“Suspicious” events within a stream of video data. Activity of interest may be defined dynamically by the Activity of interest may be defined dynamically by the

analyst during the running of the utility and flagged for analyst during the running of the utility and flagged for analysis.analysis.

28

DirectionsDirections Enhancements to the workEnhancements to the work

Working toward bridging the semantic gap and enabling Working toward bridging the semantic gap and enabling more efficient video analysismore efficient video analysis

More rigorous experimental testing of conceptsMore rigorous experimental testing of concepts Refine event classification through use of multiple machine Refine event classification through use of multiple machine

learning algorithm (e.g. neural networks, decision trees, learning algorithm (e.g. neural networks, decision trees, etc…). Experimentally determine optimal algorithm.etc…). Experimentally determine optimal algorithm.

Develop a model allowing definition of simultaneous events Develop a model allowing definition of simultaneous events within the same video sequencewithin the same video sequence

Security and PrivacySecurity and Privacy Define an access control model that will allow access to Define an access control model that will allow access to

surveillance video data to be restricted based on semantic surveillance video data to be restricted based on semantic content of video objects content of video objects

Biometrics applicationsBiometrics applications Privacy preserving surveillancePrivacy preserving surveillance

29Access Control and Access Control and BiometricsBiometrics

Access ControlAccess Control RBAC and UCON-based models for RBAC and UCON-based models for

surveillance datasurveillance data Initial work to appear in ACM SACMAT Initial work to appear in ACM SACMAT

Conference 2006Conference 2006 BiometricsBiometrics

Restrict access based on semantic content of Restrict access based on semantic content of video rather then low-level featuresvideo rather then low-level features

Behavioral type access instead of “fingerprint”Behavioral type access instead of “fingerprint” Used in combination with other biometric Used in combination with other biometric

methodsmethods

Privacy Preserving Privacy Preserving Surveillance - Surveillance -

IntroductionIntroduction•A recent survey at Times Square found 500 visible surveillance cameras in the area and a total of 2500 in New York City.

•What this essentially means is that, we have scores of surveillance video to be inspected manually by security personnel

•We need to carry out surveillance but at the same time ensure the privacy of individuals who are good citizens

31

System UseSystem Use

Raw video surveillance data

Face Detection and Face Derecognizing system

Suspicious Event Detection System

Manual Inspection of video data

Comprehensive security report listing suspicious events and people detected

Suspicious people found

Suspicious events found

Report of security personnel

Faces of trusted people derecognized to preserve privacy

System System ArchitectureArchitecture

Input Video

Breakdown input video into sequence of images

Perform Segmentation

Compare face to trusted and untrusted database

Finding location of the face in the image

Derecognize the face in the image

Raise an alarm that a potential intruder was detected

Trusted face found

Potential intruder found

33Other Applications of Other Applications of Data Mining in SecurityData Mining in Security

Intrusion detectionIntrusion detection Firewall policy managementFirewall policy management Worm detectionWorm detection Insider Threat Analysis – both network/host and physicalInsider Threat Analysis – both network/host and physical Fraud DetectionFraud Detection Protecting children from inappropriate content on the InternetProtecting children from inappropriate content on the Internet Digital Identity Management Digital Identity Management Detecting identity theft Detecting identity theft Biometrics identification and verificationBiometrics identification and verification Digital ForensicsDigital Forensics Source Code AnalysisSource Code Analysis National Security / Counter-terrorismNational Security / Counter-terrorism

34

Our Vision: Our Vision: Assured Information Assured Information

SharingSharing

PublishData/Policy

ComponentData/Policy for

Agency A

Data/Policy for Coalition

PublishData/Policy


Agency C


Agency B

PublishData/Policy

1. Friendly partners

2. Semi-honest partners

3. Untrustworthy partners

1 Data Mining for Surveillance Applications Suspicious Event Detection Dr. Bhavani Thuraisingham...

Documents

Transcript of 1 Data Mining for Surveillance Applications Suspicious Event Detection Dr. Bhavani Thuraisingham...