1 1 Bayesian Cognition Winter School at Chamonix, France 9.1.2008 Bayesian Models for Computational...

11

Bayesian Cognition Winter School at Chamonix, France

9.1.2008

Bayesian Models for Computational Laban Movement Analysis

Jörg Rett and Jorge DiasJörg Rett and Jorge Dias

22Intro: Bayesian Models for Computational Laban Movement Analysis

Laban Movement Analysis:

Model for human behaviour

Bayesian Model:

Probabilistic model to analyse

human interaction

33

ApplicationsApplications

Intro: Human Movement Analysis

Mataric et al., Socially assistive robotics for post-stroke rehabilitationJournal of NeuroEngineering and Rehabilitation, 2007

• Rehabilitation

• Socially assistive robotics

• Social robots

Analysis• Studies on

patients

44


Intro: Applications


patients

Surveillance• Public spaces

Datasets and videos of the european project caviarhttp://homepages.inf.ed.ac.uk/rbf/CAVIAR/, 2003

55


Intro: Applications


patients


Virtual Reality• Interactive virtual

worlds

Enguerran Boissier Character animation using Maya softwareLAAS/ISR Report 05, 2005

66


Intro: Applications


patients



worlds

Control Interfaces• Gesture driven

control

C. Eberst et al., Towards Programming Robots by Gestures, Test-case: Programming Bore Inspection for Small Lotsizes, ICRA, 2006

77


Intro: Skills

SkillsSkills



worlds


control

Human Motion Capture

R. Urtasun and P. Fua, 3D Tracking for Gait Characterization and Recognition, FGR, 2004

• Tracking

• Model based

• 3-D vs. 2-D


patients

88


Intro: Skills

SkillsSkillsHuman Motion Capture

P. Viola and M.J. Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, CVPR, 2001

Face Recognition



worlds


control


patients

99


Intro: Skills


Face Recognition

Hand Gesture Recognition



worlds


control


patients

• Online Behaviour

• Anticipatory Behavior

J. Rett and J. Dias: Gesture Recognition Using a Marionette Model and Dynamic Bayesian Networks (DBNs), ICIAR, 2006

1010


Intro: Skills


Face Recognition


Laban Movement Analysis



worlds


control


patients

• Expressiveness

• Semantic descriptor

J. Rett and J. Dias, Human-robot interface with anticipatory characteristics based on Laban Movement Analysis and Bayesian models, ICORR, 2007

1111


Intro: Methods


Face Recognition


Laban Movement Analysis



worlds


control


patients

MethodsMethods

Bayesian

SVD

Neural Networks

1212Laban: Major components of LMA

Space Shape

Effort

Relation-ship

Body

Laban Movement Analysis (LMA)

• Five major components• Set of semantic descriptors (labels) for movements

Method to human movements.

observedescribenotate

interprete

1313Laban: Body

Space Shape

Effort

Relation-ship

Body component

• Which body parts are moving• How is their movement is related

to the body centre (~navel).• Locomotion• Kinematics

Body

1414Laban: Space

Shape

Effort

Relation-ship

Body

Space component

• Spatial pathways of human movements inside a frame of reference• Three Axes,• Three Planes • Vector Symbols

Space

1515Laban: Effort

Space Shape

Relation-ship

Body

Effort component

• Dynamic qualities of the movement• Inner attitude towards using energy• Four bipolar Effort qualities

Space {Direct, Neutral, Indirect}

Weight {Strong, Neutral, Light}

Time {Sudden, Neutral, Sustained}

Flow {Free, Neutral, Bound}

Neutral qualities

• Single Effort: Rare, difficult to perform• Four Effort: Rare; extreme movements• Three Effort: Most natural• Two Effort: Transitions, failure

Effort

1616Laban: Effort

Space Shape

Relation-ship

Body

Action Drive (Flow = Neutral)Action Space Weight Time

Punch Direct Strong Sudden

Slash Indirect Strong Sudden

Drives

• One Effort quality is neutralEffort

1717Laban: Shape

Space

Effort

Relation-ship

Body

Shape component

• Emerging from the Body and Space components.• Focused on the body or towards a goal in space

Shape

1818Laban: Relationship

Space Shape

Effort

• Modes of interaction with oneself• … with others• … with the external environment.

Body

Relationship

1919

-80%

-70%

-60%

0%

Space

Weight

Time

Flow

Laban: Summary

Relation-ship

Assigning semantic descriptors to the movement ‘Punch’

Hands/Head

1520

2530

-50

510

15

-20

-15

-10

-5

0

5

10

ForwardHigh

Indirect

Light

Sustained

Free

Direct

Strong

Sudden

Bound

10%

50%

80%

50%

Horizontal

Vertical

Saggital

Reach Space

Spreading

Rising

Advancing

Growing

Enclosing

Sinking

Retreating

Shrinking

Shape

Effort

Space

Body

2020Design: Process of designing a Bayesian Model

?

Affirmative, now I am going to perform

this action.

Example:Human–Robot Interaction based on gestures

2121Design: Phenomenon-description

Expressive Movements

Threading a needle

Waving away bugs

Punching

Dabbing paint on a canvas

a) Describing the Phenomenon

• What is the phenomenon?

Movements can be distinguished through their expressiveness




• Which features can be observed?

Interesting objects are hands and head





• How can the features be extracted?

Using:• Commercial 3-D motion capture device• Camera based colour tracker


1015

2025-10

-50

5

-25

-20

-15

-10

-5

0

5

-10-505

-25

-20

-15

-10

-5

0

5

3-D trajectories are represented through three principal planes. a) Describing the

Phenomenon• What is the

phenomenon?



• How can the features be represented?

3-D Vertical plane


Low-level variables and their sample space.

Vector symbolsA {O, U, UR, R, DR, D, DL, L, UL}

CurvatureK {180, 135, 90, 45, 0, -45, -90, -135}

SpeedVel {Zero, Low, Medium, High}

Speed GainAcc {Zero, Low, Medium, High}







LMA variables and their relation to the movements a) Describing the

Phenomenon• What is the

phenomenon?




• How do the features relate to the phenomenon

Movement Punching

Effort.Space DirectEffort.Weight StrongEffort.Time SuddenEffort.FlowNeutral

Movement Threading a needle

Effort.Space DirectEffort.Weight LightEffort.Time SustainedEffort.FlowBound







• How do the features relate to the phenomenon

Relation of low-level variables to LMA variables

2828Design: Bayesian model


b) Building the probabilistic models

movement

vector symbols (atoms)

MI

frame

A B C

Bayes-net

• Bayes model for Space




Joint distribution

P(M I A B C )= P(M) P(I) P(A | M I)

P(B | M I) P(C | M I)





Random variables and their sample space


M {punching, ..., pointing}<n>

I {1, ..., Imax}<Imax>

A {O, F, FR, R, BR, B, BL, L, LF}<9>

B {O, U, UR, R, DR, D, DL, L, UL}<9>

C {O, U, UF, F, DF, D, DB, B , UB}<9>





• Bayes model for Effort

Space Time Weight Flow

MovementM PhPhase

E.Sp E.Ti E.We E.Fl

Bayes-net (upper part)






Bayes-net (lower part)

Space Time Weight Flow

acce-leration

velocity

curva-ture K Vel Acc

E.Sp E.Ti E.We E.Fl






• Bayes model for Phase

• Bayes model for Geometry

• Bayes model for Shape

• Connecting the sub-models

• Uncertain (Soft) evidence

Full model using Space, Effort and Phase

3434Design: Learning

a) Describing the Phenomenon.

b) Building the probabilistic model.

c) Learning of the probabilities

• What needs to be learned?

movement

atoms

MI

frame

A

Question for learning

P(A | M I)

What is the probability of a vector symbol for a movement M=m at frame I=i?






movement

atoms

MI

frame

A

Conditional Probability Table

Asking the question for all movements M and all frames I

LearningP(A | M=m1 ... mn I=1 ... imax)






• How can we learn?

Histogram learning

M = pointing I = 1

Example: Davim, trial 3

O F FR R BR B BL L FL







Zero probability problem

Some events (Atoms) have not been observed.=>Zero probability is assigned=>Problem for later classification

O F FR R BR B BL L FL







Solution:Learning based on the ‚Laplace Sucession Law‘.

An

naA a

1

P*

na number of occurences of event A=a

n number of sets

|_A_| possible values of A

3939Design: Classification


b) Building the probabilistic model


d) Defining the question for classification

Question in 2-D:

What is the probability distribution of movements m given the frame i and direction symbols of the vertical plane A?

4040Design: Classification





Question in 3-D:

What is the probability distribution of movements m given the frame i and direction symbols of all planes A, B, C?

4141Design: Continuous Update





e) Continuous update of the results

Likelihood computation

For a sequence of n observations of a.

4242Design: Continuous Update





e) Continuous update of the results

Update in 2-D:

4343Results: Confusion tables

Performance evaluation using confusion tables

6 byebye

5 pointing

4 ok

3 stretch

2 maestro

1 lunging

7 shake

8 nthrow

6 by

ebye

5 po

intin

g

4 ok

3 st

retc

h

2 m

aest

ro

1 lu

ngin

g

7 sh

ake

8 nt

hrow

Experiment 1

Using one 2-D projection (fronto-parallel view), (B atoms)

13

movem

ents

Horizontal wavingBye-bye sign

Movement 6

Testing in 13 trials of movement byebye

4444

Experiment 1


Results: Confusion tables


6 byebye

5 pointing

4 ok

3 stretch

2 maestro

1 lunging

7 shake

8 nthrow

6 by

ebye

5 po

intin

g

4 ok

3 st

retc

h

2 m

aest

ro

1 lu

ngin

g

7 sh

ake

8 nt

hrow

4 1

movem

ents

Sagittal wavingApproach sign

Movement 8

Testing in 5 trials of movement nthrow

4545

Results

Trajectories of nthrow and ok in the vertical plane



6 byebye

5 pointing

4 ok

3 stretch

2 maestro

1 lunging

7 shake

8 nthrow

6 by

ebye

5 po

intin

g

4 ok

3 st

retc

h

2 m

aest

ro

1 lu

ngin

g

7 sh

ake

8 nt

hrow

4 1

movem

ents

1015

2025-10

-50

5

-25

-20

-15

-10

-5

0

5

-10-505

-25

-20

-15

-10

-5

0

5

2025-5

05

10

-25

-20

-15

-10

-5

0

5

10

-50510

-25

-20

-15

-10

-5

0

5

10

nthrow ok

Experiment 1


4646

Final Results 2-D

95 trials

31 Wrong classifications

=> Reconition rate 67%



6 byebye

5 pointing

4 ok

3 stretch

2 maestro

1 lunging

7 shake

8 nthrow

6 by

ebye

5 po

intin

g

4 ok

3 st

retc

h

2 m

aest

ro

1 lu

ngin

g

7 sh

ake

8 nt

hrow

movem

ents

Experiment 1


4747

Experiment 2

Using the three principal planes (horizontal, vertical, sagittal), (A, B, C atoms)



6 byebye

5 pointing

4 ok

3 stretch

2 maestro

1 lunging

7 shake

8 nthrow

6 by

ebye

5 po

intin

g

4 ok

3 st

retc

h

2 m

aest

ro

1 lu

ngin

g

7 sh

ake

8 nt

hrow

5

movem

ents

Sagittal wavingApproach sign

Movement 8

Testing in 5 trials of movement nthrow

4848

Experiment 2

Using the three principal planes (horizontal, vertical, sagittal), (A, B, C atoms)



6 byebye

5 pointing

4 ok

3 stretch

2 maestro

1 lunging

7 shake

8 nthrow

6 by

ebye

5 po

intin

g

4 ok

3 st

retc

h

2 m

aest

ro

1 lu

ngin

g

7 sh

ake

8 nt

hrow

movem

entsFinal Results 3-D

95 trials



Final Results 2-D


4949Results: Confusion tables


6 byebye

5 pointing

4 ok

3 stretch

2 maestro

1 lunging

7 shake

8 nthrow

6 by

ebye

5 po

intin

g

4 ok

3 st

retc

h

2 m

aest

ro

1 lu

ngin

g

7 sh

ake

8 nt

hrow

movem

entsFinal Results 2-D perspective

95 trials



Final Results 2-D


Experiment 3

Using one 2-D projection but from a 22° rotated perspective, (B atoms)

Final Results 3-D


5050Results: Continuous classification

01

23

45

67

89

1011

12

34

56

0

0.2

0.4

0.6

0.8

i

0.91certain

quite certain

uncertain0.63

G

P(G)

i = 4i = 0 i = 5 i = 6 i = 8 i = 11

Anticipation and Certainty

5151Results: Human-Robot Interaction

Mov. 1. Demo of Nicole in Coimbra July 2006

Plant

Nicole

Godfather 2

Godfather 1

Coimbra Demo

5252Results: Human-Robot Interaction

Mov. 1. Demo of Nicole in Coimbra July 2006

5353Conclusions and Future Works

Conclusions

• We saw a process of designing computational LMA.

• The Bayesian approach was used for designing, learning and classification.

• Having online-classification opens the possibility for anticipatory behaviour.

• The Space model allows movement classification using a 2-D low-level feature.

• Better classification results are obtained by using features in 3-D.

• Under perspective variation the 2-D approach becomes worse.

Future Work

• Publication on the full Laban model including Effort and Shape.

• Publication on estimating the 3-D position from 2-D data using a geometric model.

• Test the usefulness of the social robot Nicole in a rehabilitation task.

• Extending the application of computational LMA to … … manipulatory movements (placing, grasping, etc.)… observation of human-human interaction (surveillance, etc.)

5454References

Inproceedings (Chi00Emote)Chi, D.; Costa, M.; Zhao, L. & Badler, N., The EMOTE model for Effort and ShapeSIGGRAPH 00, Computer Graphics Proceedings, Annual Conference Series, ACM Press, 2000, 173-182

Phdthesis (Zhao02Synthesis)Zhao, L., Synthesis and Acquisition of Laban Movement Analysis Qualitative Parameters for Communicative Gestures, University of Pennsylvania, 2002

Article (Zhao05Acquiring)Zhao, L. & Badler, N.I., Acquiring and validating motion qualities from live limb gesturesGraphical Models, 2005, 67, 1-16

Computational models for Laban parameters.

5555References

Laban Movement Analysis (LMA) for describing the effect of brain injuries on human movements.

Article (Foround06Changes)Foroud, A. & Whishaw, I.Q., Changes in the kinematic structure and non-kinematic features of movements during skilled reaching after stroke: A Laban Movement Analysis in two case studies, Journal of Neuroscience Methods, 2006, 158, 137-149

Theory of Labananalysis and its application to physical therapy, dance and dance therapy.

Book (Bartenieff80Body)Bartenieff, I. & Lewis, D., Body Movement: Coping with the Environment, Gordon and Breach Science, 1980

Inproceedings (Longstaff01Translating)Longstaff, J.S., Translating "vector symbols" from Laban’s (1926) Choreographie26. Biennial Conference of the International Council of Kinetography Laban, ICKL, Ohio, USA, 2001, 70-86

1 1 Bayesian Cognition Winter School at Chamonix, France 9.1.2008 Bayesian Models for Computational...

Documents

Transcript of 1 1 Bayesian Cognition Winter School at Chamonix, France 9.1.2008 Bayesian Models for Computational...