A LOOK AT AVATAR TECHNOLOGY AND SIGNING MEDIUMS

Guest Lecture for Language technology for accessibility, Institute of Computational Linguistics, University of Zurich

Robert SmithSchool of Informatics and Engineering

Institute of Technology, BlanchardstownIrish Sign Language Special Interest Group

Computational and Functional Linguistics Group

The “why” question

• Why do we need sign language?• 1st language of a nations deaf community

• Why do we need signing video or avatars?• Cannot “write” sign language like we do with spoken languages• Records• Information (TV, web, classroom, kiosk, etc…)

• Can’t Deaf people read subtitles, pamphlets etc.?• Yes but the average reading age of Deaf school leavers is comparable to

that of an 8-9 year old hearing child (Conrad, 1979)• Sign language for most Deaf people, is their 1st language

• Lip reading?• Really? Can you do it?

10/21/14 Robert Smith 2014 2

Signing mediums

•When the signer is present•Live signing

•When the signer is not present•Signing video•Traditional animation•Motion capture signing•Synthesised signing

10/21/14 Robert Smith 2014 3

The problems with video

10/21/14 Robert Smith 2014 4

Signing avatars

•Signing video

•Traditional animation•Motion capture signing•Synthesised signing

10/21/14 Robert Smith 2014 5

These methods require knowledge of animation

The reminder of this lecture is dedicated to understanding avatar animation

Origin of AvatarsAvatar (Original, Hindu term), manifestation or

appearance of a Hindu deity

Avatar (computing term), the graphical representation of a user

Avatar (Sign Language term), a 3D animated character used to deliver Sign Language utterances

Robert Smith 201410/21/14 6

ANIMATION BASICS

10/21/14 Robert Smith 2014 7

Frame on movie reel

Robert Smith 201410/21/14 8

10/21/14 Robert Smith 2014 9

Zoopraxiscope

Animated ball

Animation Frame

Robert Smith 201410/21/14 10

2D animation

Robert Smith 201410/21/14 11

2D Evolution, Pencil-to-Pencil

Robert Smith 201410/21/14 12

3D ANIMATION

10/21/14 Robert Smith 2014 13

2D - 3D animation

1994, and the first completely computer-generated animated movie was Toy Story

Robert Smith 201410/21/14 14

Concept drawing

Body shape, sizes, colours, textures etc…

Robert Smith 201410/21/14 15

Analogy (Papier-mâché)

1. Structure 2. Mesh3. Texture4. Colour

10/21/14 Robert Smith 2014 16

Analogy (Papier-mâché)

1. Structure 2. Mesh3. Texture4. Colour

10/21/14 Robert Smith 2014 17

We will look at this last

Analogy (Papier-mâché)

1. Structure 2. Mesh3. Texture4. Colour

10/21/14 Robert Smith 2014 18

This 1st

Body Shape - 3D polygon mesh

Robert Smith 201410/21/14 19

Mesh structure

Robert Smith 201410/21/14 20

Number of polygons is important

Robert Smith 201410/21/14 21

Adding skin detail

Normal maps (skin pores etc)

Specular and diffuse lighting mapsBump maps / texture maps

Robert Smith 201410/21/14 22

Bump/Specular/Normal maps on a 3D mesh

Robert Smith 201410/21/14 23

Example

10/21/14 Robert Smith 2014 24

Analogy (Papier-mâché)

1. Structure 2. Mesh3. Texture4. Colour

10/21/14 Robert Smith 2014 25

Structure/Movement (armatures)

Robert Smith 201410/21/14 26

Body rigging

Robert Smith 201410/21/14 27

Basic facial rigging

Robert Smith 201410/21/14 28

Robert Smith 201410/21/14 29

Robert Smith 201410/21/14 30

3D Rendering

Robert Smith 201410/21/14 31

Hardware vs time

10/21/14 Robert Smith 2014 32

Real-time rendering

3D Rendering - examples

Robert Smith 201410/21/14 33

3D animation methods

Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…

1. ‘Traditional’ Animation using animators

2. Motion capture (mocap)

3. Full synthesis

Robert Smith 201410/21/14 34

3D animation methods

Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…

1. ‘Traditional’ Animation using animators

2. Motion capture (mocap)

3. Full synthesis

Robert Smith 201410/21/14 35

Using Animators

• 3 year production

• Teams of artists and animators

• 3500 Sun Microsystems processors, compared with 1400 for Toy Story 2 and only 200 for Toy Story

Robert Smith 201410/21/14 36

Signed Language Example

Robert Smith 201410/21/14 37

3D animation methods

Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…

1. ‘Traditional’ Animation using animators

2. Motion capture (mocap)

3. Full synthesis

Robert Smith 201410/21/14 38

Motion captureThe most commonly used technique for sign language animation on the web is motion capture

Robert Smith 201410/21/14 39

Motion capture setup

Robert Smith 201410/21/14 40

Facial emotion on these projects was good

Motion/Performance capture - examples

Robert Smith 201410/21/14 41

“Avatar” movie• 190 inferred cameras + Dozens of HD cameras

Robert Smith 201410/21/14 42

Facial mapping

Robert Smith 201410/21/14 43

360˚ HD photo sphere

Robert Smith 201410/21/14 44

Capturing emotion

Robert Smith 201410/21/14 45

MoCap drawbacks

• Expensive equipment.

• Requires labour-intensive calibration of the equipment.

• …and post-editing of the captured data.

• Good for the movie industry• Realism, faster production, cheaper

• Bad for Sign Language• Finite data set, Signed Exact English

Robert Smith 201410/21/14 46

3D animation methods

Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…

1. ‘Traditional’ Animation using animators

2. Motion capture (mocap)

3. Full synthesis

Robert Smith 201410/21/14 47

SL Synthesis

• Sign synthesis is not new (The concept is over 10 years old).

• Less costly to implement.

• Has the flexibility of a writing system.

• Real-time rendering.

Robert Smith 201410/21/14 48

SL Synthesis2005-2006, ASL project,

University of Pennsylvania

2006-2007, The South African SL Project,

University of Stellenbosch

2004-2007, The Spanish SL project,

University of Madrid

Robert Smith 201410/21/14 49

Issues with synthesis

• Focus is often narrowed i.e.• Machine Translation• Specific task• Specific linguistic attribute

• Low processing requirements for rendering• Mesh polygons• Armature bones• Texture/Specular/Normal mapping

• Little work to date on NMF

Robert Smith 201410/21/14 50

State-of-the-art system ( )

HamNoSys-to-SiGML

AnimGenAnimation data

SiG

ML

HamNoSys

<sigml> <hns_sign gloss="$PROD:Morning"> …al> </hns_sign></sigml>

Robert Smith 201410/21/14 51

SummaryMoCap

Synthesis

Concept Mesh Skin Rigging Render

HamNoSysAdd animation

parametersMachine readable

Render

Concept Mesh Skin Rigging Render

Capture Motion Data

SearchData

Machine readable

Assemble utterance

Add animation parameters Render

Robert Smith 201410/21/14 52

Summary

•When the signer is present•Live signing

•When the signer is not present•Signing video•Traditional animation•Motion capture signing•Synthesised signing

10/21/14 Robert Smith 2014 53

Questions

?

Robert Smith 201410/21/14 54