Visual Event Visual Event Detection & Detection & RecognitionRecognition

Filiz Bunyak Ersoy, Ph.D. Filiz Bunyak Ersoy, Ph.D. studentstudent

Smart Engineering Systems Smart Engineering Systems LabLab

Research InterestResearch Interest

Recognition of visual events from video Recognition of visual events from video sequences. sequences.

Use of visual event information for intelligent Use of visual event information for intelligent video surveillance and semantic video video surveillance and semantic video indexing & retrieval.indexing & retrieval.

Incorporation of learning to event modeling Incorporation of learning to event modeling and recognition.and recognition.Very few event detection systems that are currently available Very few event detection systems that are currently available are not flexible, work for a very limited domain for very limited are not flexible, work for a very limited domain for very limited number of predefined, mostly hand-coded events. They are not number of predefined, mostly hand-coded events. They are not designed to be extended or modified. Learning will enable designed to be extended or modified. Learning will enable adaptability and extensibility of an event detection systemadaptability and extensibility of an event detection system..

Content of Video DataContent of Video Data

Low Level Low Level

Visual Visual FeaturesFeatures

ColorColor TextureTexture ShapeShape MotionMotion Shot Shot

BoundariesBoundaries

Mid LevelMid Level

Semantic Semantic ContentContent

People/People/ObjectsObjects

LocationLocation ActionsActions TimeTime

High LevelHigh Level

Semantic Semantic ContentContent

StoryStory ConceptConcept EventEvent

ChallengesChallenges1-Representation:1-Representation: Modeling visual Modeling visual

features features Image/video object Image/video object

representation representation Description of spatio-Description of spatio-

temporal relationships temporal relationships High-level event High-level event

representationrepresentation2-Analysis: 2-Analysis: Segmentation of video Segmentation of video

objects objects Adaptive grouping of Adaptive grouping of

features & objects features & objects Compressed-domain Compressed-domain

feature extraction feature extraction

3-Indexing: 3-Indexing: Efficient indexing Efficient indexing

algorithms for high-algorithms for high-dimensional feature dimensional feature space space

Robust, scalable Robust, scalable indexing algorithms for indexing algorithms for spatio-temporal queries spatio-temporal queries

4-Summarization:4-Summarization: Automated Automated

summarization of visual summarization of visual contentcontent

Visualization of content Visualization of content at different levels at different levels

Proposed FrameworkProposed Framework

Feature Extraction

Motion Analysis

Event Inference

ObjectClassification

ObjectsRelationships

Events

ContextObject, Scene &Event Libraries

Context is any a priori informationprovided to the system.

Events

Object Information

Object Trajectories & Spatio-temporal relationships

Motion AnalysisMotion Analysis

Moving Object DetectionMoving Object Detection Temporal differencingTemporal differencing Background subtractionBackground subtraction Optical FlowOptical Flow

TrackingTracking Region-based methodsRegion-based methods Contour-based methodsContour-based methods Feature-based methodsFeature-based methods Model-based methodsModel-based methods

Moving ObjectDetection

Feature Extraction

CorrespondenceAnalysis

Prediction

Update

Context

Object States

Tracking

Single-view methods Single-view methods (single camera)(single camera)

Multi-view methods Multi-view methods (multiple cameras)(multiple cameras)

Event InferenceEvent InferencePossible Event Inference Possible Event Inference

MethodsMethods Rule based Rule based Logical FormalismsLogical Formalisms

Temporal LogicTemporal Logic Event LogicEvent Logic Fuzzy LogicFuzzy Logic

Bayesian belief networkBayesian belief network Hidden Markov modelHidden Markov model Petri-netPetri-net GrammarGrammar

Event Inference Methods

InformationAbout Objects

Spatio-temporal Relationships

A priori InformationAbout the Application, Goal, Scene, Objects. Event descriptions.

EVENTS

Applications of Event Applications of Event Detection/Recognition Detection/Recognition

Surveillance and MonitoringSurveillance and Monitoring Traffic (track vehicle Traffic (track vehicle

movements and annotate movements and annotate action in traffic scenarios.)action in traffic scenarios.)

Detection of accidents, traffic Detection of accidents, traffic violations, congestions.violations, congestions.

Gather statistics about human Gather statistics about human activities, road utilization etc.activities, road utilization etc.

Surveillance of public places / Surveillance of public places / shops / offices etc. shops / offices etc.

Detection of atypical Detection of atypical incidents, theft, vandalism, incidents, theft, vandalism, shoplifting, abandoning shoplifting, abandoning (possibly dangerous) objects.(possibly dangerous) objects.

Indexing of Broadcast Indexing of Broadcast VideoVideo Sports video indexing for Sports video indexing for

newscasters or trainers.newscasters or trainers. Semantic indexing for Semantic indexing for

automated annotation for automated annotation for content retrieval.content retrieval.

Interactive environments:Interactive environments: environment that respond to environment that respond to the activity of occupants.the activity of occupants.

Robotic collaboration:Robotic collaboration: creating robots that can creating robots that can effectively navigate their effectively navigate their environment and interactenvironment and interact with other people and robotswith other people and robots..

Some Interesting Systems Some Interesting Systems from the Industry-1from the Industry-1

Realtime Video Analysis Realtime Video Analysis Group @ Siemens Group @ Siemens Corporate ResearchCorporate Research

Subway monitoring Subway monitoring SystemSystem

Real time segmentation of Real time segmentation of people in subway people in subway platforms for the purpose platforms for the purpose of congestion (crowding) of congestion (crowding) detectiondetection

Honeywell Laboratories:Honeywell Laboratories:

Cooperative Camera Network Cooperative Camera Network (CCN): Indoor(CCN): Indoor

Reports the presence of visually Reports the presence of visually tagged individual throughout a tagged individual throughout a building structure. building structure.

Meant to be used for Meant to be used for monitoring potential shoplifters monitoring potential shoplifters in department stores.in department stores.

Detection of Events for Threat Detection of Events for Threat Evaluation & Recognition Evaluation & Recognition (DETER): Outdoor(DETER): Outdoor

Monitor large open spaces like Monitor large open spaces like parking lots and reports parking lots and reports unusual moving patterns by unusual moving patterns by pedestrians & vehicles.pedestrians & vehicles.

Some Interesting Systems Some Interesting Systems from the Industry-2from the Industry-2

Mitsubishi Electric Mitsubishi Electric Research Research LaboratoriesLaboratories

ApplicationsApplications Detecting accidents through Detecting accidents through

analysis of traffic surveillance analysis of traffic surveillance video. video.

Detection of traffic jams using Detection of traffic jams using our MPEG-7 motion activity our MPEG-7 motion activity descriptor. descriptor.

Extraction of semantic features Extraction of semantic features from low-level features of soccer from low-level features of soccer games. games.

http://www.merl.com/projects/event-http://www.merl.com/projects/event-detection/detection/

Some Interesting Systems Some Interesting Systems from the Industry-3from the Industry-3

ASCOM: INVIS Traffic ASCOM: INVIS Traffic DetectDetect Traffic speed and traffic Traffic speed and traffic

flow density        flow density        Congestion and slow Congestion and slow

traffic        traffic        Stationary vehicles Stationary vehicles

(possibly an accident)(possibly an accident)       

Copyright © 2002 Ascom Copyright © 2002 Ascom www.ascom.comWrong Way Early Warning

System Recognizes vehicle patterns and

compares subsequent images to determine vehicle and direction.

Thus it can detect any car driving the wrong way into a lane against the flow.