Real-Time Human Posture Reconstruction in Wireless Smart Camera Networks

Chen Wu, Hamid Aghajan Wireless Sensor Network Lab, Stanford University, USA

IPSN 2008

Speaker Lawrence

Outline

• Background• Motivation• Goal• Challenge• Strategy for Camera Sensor Network• System Overview• Wireless Smart Camera (Hardware)• Human Pose Estimation (Algorithm)• Result• Conclusion

Background

• Traditional Camera network for surveillance & security

• New applications of camera network for multimedia, video conference…etc

• Wireless Camera network – Scalability– Privacy preservation– Flexibility on collaboration scheme between cameras

Motivation

• As pervasive sensors the cameras can free the users from wearable devices.

• Lack of real-time vision algorithm to achieve moderate complexity, robustness and scalability.

Goal

• Implementation of human pose interpretation on a wireless smart camera network.

• Employing distributed processing– Real-time vision & scalability for complex vision algorithms.

Challenge

• A vision sensor network poses three key challenges:

– High computation capacity for real-time performance.

– Wireless links limit image transmission (bandwidth & energy)

– Lack of established vision-based fusion mechanisms (by real time)

Strategy for Camera SN

• Difference between Camera network & Distributed vision processing strategy systems.– Employ cameras as a wireless sensor network.

• Strategy:1. Video data reducing (Network bandwidth)2. Level of vision analysis to different PHY processors

Strategy for Camera SN (cont.)

Central PC

SmartCamera

Level of vision analysis to different PHY processors

Scalability : Spatial and functional parallelism

• Each camera video processes its own data(spatial)• Running their own function modules(functional)

Strategy for Camera SN (cont.)

Smart camera communicate with the central PC through ZigBee

System Overview

LCD display

Smart camera

Different ZigBee channels

Data flow in the system

System Overview (cont.)

Semaphore tech for DPRAM

P.S. DPRAM allows multiple r or w to occur at the same time.

Asynchronous

Wireless Smart Camera

• Hardware Platform– VGA color image

sensor– SIMD

processor(IC3D)– Embedded

processor(8051)– ZigBee platform

• Parallel arch power consumption

• LPA(320 PEs) data processing

• GCP control IC3D & DSP operations

• PE # video format, e.g., VGA(640*480)

• The main design factors of SIMD frequency & PE #

Wireless Smart Camera (cont.)

MP-SIMD

• Data sharing between processors– PDRAM functions as an asynchronous connection

between IC3D and 8051– Semaphore tech to prevent mutual access

• Wireless communication– P2P structure offers direct camera to PC communication– Maximum data rate : 100 Kbit/sec

Wireless Smart Camera (cont.)

• Review (Algorithm)– Goal : 2D to 3D– Ambiguity: Perspective views of the camera or self-

occlusion of human body

• Pose Estimation Approach– Top-down– Bottom-up

Human Pose Estimation

• Top-down

Human Pose Estimation (cont.)

• Bottom-up

Human Pose Estimation (cont.)

Top-down vs Bottom-up• Top-down

– Strength• Occlusion handling• Contours & body

part association

– Weakness• Search tech

complexity(depth)• Computational

load(projection)

• Bottom-up– Strength

• Much less demands on 3D switch

– Weakness• Complex assemble• Difficult to detect

occlusions

Human Pose Estimation (cont.)

• Challenges & Method– Bandwidth constraint

(100Kbits/sec)/(30frames/sec)/(8bits/Byte) ≈ 400B/frame solution: Detect body part cancroids coordinates– Limited image processing capability of the SIMD

processorsolution: Color segmentation

– Robustness with varied environment solution: Auto-balancing

filtering & combination

Human Pose Estimation (cont.)

In-node processing

• Detect positions(x, y):– Head, shoulders and hands– 2Bytes for x and y

• Detect mechanism:– Face -> face color model– Head -> skin color model– Shoulders -> shirt color model (low-pass filter)

Human Pose Estimation (cont.)

Human Pose Estimation (cont.)

The image processing program on IC3D

Human Pose Estimation (cont.)

• Processing on the central PC

– Noise filtering and 2D to 3D reconstruction

Human Pose Estimation (cont.)

• Demo: Virtual ball-playing game

– Demo Video

Results

Standard Deviation of detected body part coordinates in the smart cameras (in pixels) and those after noise filtering

Results (cont.)

Demo

Original data from the smart cameras and data after noise filtering

Head

Left shoulder Right shoulder

Results (cont.)

Results (cont.)

Left hand Right hand

Conclusion

• Propose an algorithmic strategy to approach vision problems in a wireless camera sensor network

• Major aspect of the strategy:– reduce video data locally through smart camera

• Implement a prototype system of 3D human reconstruction using a wireless smart camera.

• Wireless camera networks will offer potentials for user-centric applications.

Thanks for listening