Fine-grained Walking Activity Recognition via Driving Recorder Dataset

Hirokatsu KATAOKA, Yoshimitsu AOKI†, Yutaka SATOH Shoko, OIKAWA‡, Yasuhiro MATSUI‡

National Institute of Advanced Industrial Science and Technology (AIST) † Keio University

‡ National Traffic Safety and Environment Laboratory (NTSEL)

http://www.hirokatsukataoka.net/

Background •  ADAS; Advanced Driver Assistance Systems –  A large amount of technologies have been proposed –  The pedestrian deaths are on the rise –  Detection systems, environment, autonomous driving car

@Pedestrian  and  vehicle  detec0on   @Lane  detec0on  (Environment  understanding)  

@Autonomous  driving  in  Google  

ADAS technologies are highly required!

Pedestrian detection •  Vision-based detection is one of the important techniques –  Pedestrian detection survey [Benenson+, ECCVW2014] •  They implemented and compared 40+ detection approaches

–  Deep Learning is applied to detect pedestrians [Sermanet+, CVPR2013] •  Convolutional neural networks (CNN) •  Automatic feature training and classifier

Better

Detection rate has been improving

New step toward “pedestrian analysis” •  High-performance pedestrian localization –  Task-assistant CNN (TA-CNN) [Tian+, CVPR2015] •  The framework is consist of CNN feat. & attribute (e.g. background, location)

•  Limitations of pedestrian safety systems –  Pedestrian detection at present –  Detection range: width of the vehicle

Going to the next “pedestrian analysis” researches!

Motivation •  Fine-grained pedestrian activity recognition in addition to pedestrian detection –  More detailed activity analysis –  Pedestrian activity intention understanding

Probability map of danger

1.0 second is crucial time in ADAS

Why fine-grained?

Walking along a sidewalk

Turning

Crossing a roadway

Process flow •  Fine-grained walking activity recognition

1.  Pedestrian localization 2.  Activity analysis

Improved dense trajectories (iDT)

Pedestrian detection

x x x x x x x x x x x x x x x

x x x

Trajectory (in t + L frames)

Feature extraction (HOG, HOF, MBH, Traj.)

Bag-of-words (BoW)

iDT

Detection system •  Per-frame CNN feature and NMS –  Region of interesting (ROI) –  VGGNet feature in the detection problem –  Non-maximum suppression for combining detection windows

・・・~  

~・・・  

NMS

Activity Recognition •  Improved Dense Trajectories (iDT) [Wang+, ICCV2013] –  Pyramidal image sequences and flow tracking –  Feature descriptors on trajectories –  Feature representation with bag-of-words (BoW)

Walking Crossing Turning

Experiments •  Fine-grained walking activity recognition –  Understanding small changes while people walking •  Walking along a side walk & Crossing a road way •  Walking straight & turning •  Walking & riding a bicycle

(a)  crossing (b)  walking (c)  turning (d)  bicycle

Datasets and implementations •  NTSEL dataset & Near-miss dataset

•  Implementation –  Localization: VGGNet layer-pooling-5 –  Feature: IDT (HOG, HOF, MBH, Traj.) –  Classifier: Support vector machine (SVM)

(a)  crossing (b)  walking (c)  turning (d)  bicycle

NTSEL dataset Near-miss DR dataset

http://www.jsae.or.jp/hiyari/0907/

Results •  On the NTSEL and Near-miss DR dataset

Descriptor % on NTSEL % on Near-miss DT (Traj.) 76.5 77.9 DT (HOF) 93.7 75.9 DT (HOG) 85.6 76.4 DT (MBHx) 87.7 59.3 DT (MBHy) 86.7 60.8

–  Outstanding performance rate with IDT 93.7% on NTSEL and 77.9% on Near-miss DR dataset

Spatio-temporal analysis •  Using iDT, temporal direction is analyzed –  Fewer frames are better in the space-time –  Sudden motion should be recognized

Demonstration •  Fine-grained ped. activity recognition on NTSEL dataset –  Improved Dense Trajectories (93.7%)

Conclusion •  Fine-grained walking activity analysis for the new step of pedestrian intention understanding –  State-of-the-art motion analysis algorithms are implemented –  High-performance localization and recognition on the traffic datasets –  Pedestrian analysis are executed in detail

•  More flexible models and intention understanding –  We need more data in learning step –  Transition model or more strong temporal feature should be implemented