Toward Real-Time Extraction of Pedestrian Contexts with Stereo Camera

Kei Suzuki, Kazunori Takashio, Hideyuki Tokuda,

Masaki Wada, Yusuke Matsuki, Kazunori Umeda

Graduate School of Media and Governance, Keio University

Department of Precision Mechanics, Faculty of Science and Engineering, Chuo University

Fifth International Conference on Networked Sensing SystemsJune 17 - 19, 2008 // Kanazawa, Japan

Introduction

• Vision-based monitoring systems are becoming important

• Many researches of pedestrian recognition from vision data have been conducted.– Pedestrian Detection, Tracking, Activities

Security camera

Motivation

• Current works are difficult for extracting pedestrian’s high-level contexts.

Pedestrian Detection, Tracking, Activity

Pedestrian high-levelContexts

How suspicious is this situation?

SnatchSkirmish

Goals

• Extract the high-level contexts of pedestrians from vision data in real-time– We aim to extract suspicious individuals, groups

and uneasy atmosphere based on the position, velocity, width, and height of those individuals.

– We developed a prototype system that can find people tumbling, walking in a group.

At a street corner system image

Our Approach

• Use of a stereo camera system that focuses on a moving region– We can make real-time 3D measurement of more

than one pedestrian’s region• center of gravity coordinates, height, width, label number

– Easy to install in a new place

• Infer the pedestrian contexts by using Bayesian Networks– We model pedestrian movement as time-series data– We make Bayesian models and let them learn for

each context

The target contexts table

individual

group

Atmosphereof the place

Prototype system

Not yet

walking in a group

tumble

Project target

WalkingRunningTumble

SnatchSkirmish

CrowdedQuiet

System Overview

• Hardware architecture

At a street corner

Stereo cameraAnalyze vision data

network

Stereo camera system Pedestrian context Infer system

Display result of contexts

Infer the context

System Overview

• The flow of inferring contexts

e.g.A: velocity vector similarityB: the average distanceC: the average vector angle

The thread of event detection

The inference thread using theBayesian Networks

sliding window

Input variables into Bayesian Model

Stereo camera system data

Experiment ． Extract contexts with real-data

• Extract two pedestrian contexts– Tumble as a individual context– Walking with friends as a group

context

• Result– Show the effectiveness of extracting

two or more contexts in real-time.– The required time of extracting the

contexts was 58 msec, but it worked in real-time due to the event trigger model.

Conclusion / Future Work

• We extracted the pedestrian contexts with stereo camera system

• We developed the prototype system, and confirmed by the experiments.

• Future work– Further evaluation of accuracy, and compare

its performance with some other methods.

• Thank you!

Experiment1 ． Stereo camera’s output test

• Walk 6.5m from the side of a camera, then turn right.

-1

0

1

2

3

4

5

6

7

1 4 7 1013 1619 2225 2831 3437 4043 4649 5255 5861 6467 70

xyz

Stereo camera’s frame:14 frame/sec

Camera

Dis

tanc

e fr

om c

amer

a (m

)

Time (sec)

Tur

n rig

ht

Bayesian Network Model

• The sample of tumble model

Pedestrian Recognition at a street corner

• Many projects of pedestrian recognition from video data have been conducted.– Pedestrian Detection– Pedestrian Tracking– Pedestrian’s activities

Security camera

Characteristics of Pedestrian

• The activities of Individuals– walking, running, tumbling

• The activities of Groups, mobs– Harmless group

• Companion ， with the same intention

– Contingent group• Moving to same direction, temporary crowded with people

– Suspicious group• Snatchers, fighting, entering no admittance area

• Atmospheres of a place– Crowded, quiet

Pedestrians have 3 characteristics based on their movement.

Goals

• Extracting from suspicious individuals and uneasy atmosphere by using video cameras – We aim to extract high-level pedestrian

contexts in real-time– The system infers contexts based on

pedestrians’ moving region data

Stereo Camera System that focusing on moving regions

• Calculate Moving region feature– Center of gravity coordinate– Distances from camera– Height and width– Timestamp– Label number

Project abstract (1/2)

• Extracting from suspicious individuals and uneasy atmosphere by using stereo cameras – We aim to extract high-level pedestrian contexts in

real-time– The system infers contexts based on pedestrians’

moving region data from stereo camera

At a street corner

Project abstract (2/2)• Recognizing group and mobs

– harmless crowd• Companion group

– Accidental mobs• spectators

– Suspicious mobs target⇒• snatchers, fighting mobs

Bayesian Network

• Extracting of high-level pedestrian contexts using Bayesian Network.

System Over View

At the street

Stereo cameraVideo data analyze

network

Stereo camera system Pedestrian context Infer system

Display result of contexts

Contextinfer

Hardware architecture

Pedestrians at street corner

Stereo cameraProcessing videodata

network

Stereo camera system Inferring Pedestrian Contexts in real-time

Contexts inferred result

Infering contexts

Prototype System’s target Contexts

target

individuals

group

AtmosphereOf a place

Prototype system’s target contexts

Not yet

Walking with friends

tumble

Project target contexts

Normal walkingRunningTumble

Snatchers,Fighting

CrowdingQuiet

実験 2 ．コンテクスト推定部の負荷実験

• コンテクスト推定の最大負荷時

コンテクスト同時推定数 平均推定時間 CPU使用率

2 40msec 100%4 160msec 100%