Motion Pictures and Mathematics Motion Pictures and Mathematics

Chris BreglerChris Bregler

Courant InstituteCourant Institute

New York UniversityNew York University

Turing Test: Artificial IntelligenceTuring Test: Artificial Intelligence

Cogs:Cogs: Rodney Brooks Rodney Brooks

Simulate HumanSimulate Human

Turing Test: Computer GraphicsTuring Test: Computer Graphics

Create Illusion Create Illusion On ScreenOn Screen

Final FantasyFinal Fantasy

Mathematical ModelsMathematical Models

• Lighting

• Skin

• Hair

• Water

• Cloth

• Primary Human Motion ?? (Simulate Brain)

(Spider Man II, Debevec et al)

(Van Jensen et al)

(ILM, Fedkiw et al)

Two leading techniques for AnimationTwo leading techniques for Animation

Performance-Capture:Mocap / Rotoscoping

Traditional:Hand-Animated

Pixar / Disney Electronic Arts

NYU Motion Capture LabNYU Motion Capture Lab

QuickTime™ and aCinepak decompressor

are needed to see this picture.

NYU Motion Capture LabNYU Motion Capture Lab

QuickTime™ and aSorenson Video 3 decompressorare needed to see this picture.

““Satan’s Rotoscope” ControversySatan’s Rotoscope” Controversy

Performance-Capture:Mocap / Rotoscoping

Traditional:Hand-Animated

Pixar / Disney Electronic Arts

Disney: Expressive MotionDisney: Expressive Motion

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Motion ReferenceMotion Reference

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Rotoscope / Mocap: HistoryRotoscope / Mocap: History

• Disney: Helen Stanley Sleeping Beauty

Mathematical Mathematical MotionMotion Models Models

Robot Skeleton Model

Human Skeleton Model

Non-Rigid Model

Style Model

Motion Model: Kinematic Chain Motion Model: Kinematic Chain

Motion Capture = Math. EquationMotion Capture = Math. Equation

10 = a * 5

Optical Flow Kinematic Chains Optical Flow Kinematic Chains

M *=[v1,v2,…,vn]

Motion Capture = Least Squares Sol. Motion Capture = Least Squares Sol.

M *=

?

[v1,v2,…,vn]

Eadweard Muybridge Eadweard Muybridge

Tracking Kinematic Chains Tracking Kinematic Chains

QuickTime™ and aCompact Video decompressorare needed to see this picture.

Mathematical Motion ModelsMathematical Motion Models

Robot Skeleton Model

Human Skeleton Model

Non-Rigid Model

Style Model

Mathematical Motion ModelsMathematical Motion Models

Robot Skeleton Model

Human Skeleton Model

Non-Rigid Model

Style Model

Humans have no fixed axis Humans have no fixed axis

The Digital Human Project

Walt Hyneman, Sony / Disney

Stuart Sumida, Cal State San Bernardino

Elizabeth Rega, Western Univ. of Health Science, Pomona

Humans have no fixed axisHumans have no fixed axis

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Optical Flow Kinematic Chains Optical Flow Kinematic Chains

M*=[v1,v2,…,vn]

?

Also estimate Model Also estimate Model

M *=

? ?

[v1,v2,…,vn]

Estimate Kinematics Estimate Kinematics

Kathy Pullen

QuickTime™ and aCompact Video decompressorare needed to see this picture.

Mathematical Motion ModelsMathematical Motion Models

Robot Skeleton Model

Human Skeleton Model

Non-Rigid Model

Style Model

Mathematical Motion ModelsMathematical Motion Models

Robot Skeleton Model

Human Skeleton Model

Non-Rigid Model

Style Model

-

Acquisition of Acquisition of NonrigidNonrigid 3D Kinematics 3D Kinematics

WithGene Alexander, Henning Biermann, Aaron Hertzmann, Lorenzo Torresani, Danny Yang, Hrishi Deshpande

3D Shape Model 3D Shape Model

Linear Interpolation between 3D Key-Shapes:

S1 S2S

S = 40% S1 + 60% S2

Basis Shape FactorizationBasis Shape Factorization

Complete 2D Tracks or Flow Matrix-Rank <= 3*K

M*= [v1,v2,…,vn]

Nonrigid 3D Kinematics from point tracks Nonrigid 3D Kinematics from point tracks

- QuickTime™ and a

YUV420 codec decompressorare needed to see this picture.

-

Nonrigid 3D Kinematics results Nonrigid 3D Kinematics results

-

Nonrigid 3D Kinematics results Nonrigid 3D Kinematics results

-

Nonrigid 3D Kinematics results Nonrigid 3D Kinematics results

Motion Capture

Modeling

Synthesis

From Pixels to 3D Blend Shapes From Pixels to 3D Blend Shapes (Torresani et al)(Torresani et al)

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Motion Capture

Modeling

Synthesis

From Pixels to 3D Blend Shapes From Pixels to 3D Blend Shapes (Torresani et al)(Torresani et al)

QuickTime™ and aCinepak decompressor

are needed to see this picture.

CHANGING Motion (Facial Speech)CHANGING Motion (Facial Speech)

Video Rewrite

Bregler, Slaney, Covell

CHANGING AgeCHANGING Age

Disney Feature Animation: Gemeni Man

Hoyt Yeatman, Director / Visual Effects SupervisorJinko Gotoh, ProducerAndrew Millstein, Exec. ProducerLance Williams, Technical ArchitectPrice Pethel, Creative Director / ActorWalter Hyneman, Digital Effects SupervisorShari Judson, Production Coordinator, Karla Ventocilla, Production Asst.Hioki Itokazu, ModelerXinmin Zhao, John Lewis, Sean Jenkins, Dan Ruderman, Eric Enderton, Chris Bregler, Jim Rothrock, Rasmus Tamstorf, Heather Pritchett,Shyh-Chyuan Huang, DevelopersSergi Sagas, Jason MacLeod, Manuel Kraemer, Lewis Siegel, Kyle StrawitzRichard Lehman, Technical Directors

Disney Face ProjectDisney Face Project

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

Mathematical Motion ModelsMathematical Motion Models

Robot Skeleton Model

Human Skeleton Model

Non-Rigid Model

Style Model

Mathematical Motion ModelsMathematical Motion Models

Robot Skeleton Model

Human Skeleton Model

Non-Rigid Model

Style Model

Turning to the Masters:Turning to the Masters:

Motion Capturing CartoonsMotion Capturing Cartoons

Chris Bregler, Lorie Loeb, Erika Chuang, Hrishi DeshpandeChris Bregler, Lorie Loeb, Erika Chuang, Hrishi Deshpande

Stanford UniversityStanford University

[CB: Other picture coming]

Turning to the Masters:Turning to the Masters:

Expressive AnimationsExpressive Animations

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Turning to the Masters:Turning to the Masters:

Expressive AnimationsExpressive Animations

QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.

ExamplesExamples

© Warner Bros.

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Mathematical Motion ModelsMathematical Motion Models

Robot Skeleton Model

Human Skeleton Model

Non-Rigid Model

Style Model

Laban CaptureLaban Capture

• With Peggy Hackney and Ted Warburton and Alyssa Lees and Lorenzo Torresani

• Laban Movement Analysis• Capturing Effort• Data Driven

CHANGING Motion (Style) “Effort Space”CHANGING Motion (Style) “Effort Space”

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

Free & Light Bound & Strong

Torressani et al

CHANGING Motion (Style) “Effort Space”CHANGING Motion (Style) “Effort Space”

Normal 125% Peggy

Lees et al

QuickTime™ and aCinepak decompressor

are needed to see this picture.

QuickTime™ and aCinepak decompressor

are needed to see this picture.

CHANGING Motion (Facial Speech)CHANGING Motion (Facial Speech)

Video RewriteInterval Corp.

Bregler, SlaneyCovell

CHANGING Motion (Body Speech)CHANGING Motion (Body Speech)

M. Stone et alRutgers, NYU,Electronic Arts

CHANGING AgeCHANGING Age

Disney Feature Animation: Gemeni Man

Hoyt Yeatman, Director / Visual Effects SupervisorJinko Gotoh, ProducerAndrew Millstein, Exec. ProducerLance Williams, Technical ArchitectPrice Pethel, Creative Director / ActorWalter Hyneman, Digital Effects SupervisorShari Judson, Production Coordinator, Karla Ventocilla, Production Asst.Hioki Itokazu, ModelerXinmin Zhao, John Lewis, Sean Jenkins, Dan Ruderman, Eric Enderton, Chris Bregler, Jim Rothrock, Rasmus Tamstorf, Heather Pritchett,Shyh-Chyuan Huang, DevelopersSergi Sagas, Jason MacLeod, Manuel Kraemer, Lewis Siegel, Kyle StrawitzRichard Lehman, Technical Directors

Disney Face ProjectDisney Face Project

Scaling UpScaling Up

PixelsPixels

1000’s 1000’s PeoplePeople

Scaling UpScaling Up

Largest Mocap Volume so far, 50x50x10 feet, 6 “players”

Electronic Theater this year in LA: 190x190x40And we have more “players”: 4000

© Motion Analysis for “Nike Gamebreakers”

SIGGRAPH Electronic TheaterSIGGRAPH Electronic Theater

The Core TeamThe Core Team

Sally RosenthalSally Rosenthal

Katie Salen, Parsons School of DesignKatie Salen, Parsons School of Design

Luke Dubois, Cycling74 / NYU-ITPLuke Dubois, Cycling74 / NYU-ITP

Tom Igoe, NYU-ITPTom Igoe, NYU-ITP

Michael Naimark, USC / NYU-ITPMichael Naimark, USC / NYU-ITP

Jon Meyer, NYU-MRLJon Meyer, NYU-MRL

Jeremi Sudol, NYU-MRLJeremi Sudol, NYU-MRL

Akex Postelnicu, RPIAkex Postelnicu, RPI

Michael Rabinovich, RPIMichael Rabinovich, RPI

Kevin Feeley, NYU-MRLKevin Feeley, NYU-MRL

Clilly Castiglia, NYU-MRLClilly Castiglia, NYU-MRL

Bo Wright, NYU-MRLBo Wright, NYU-MRL

Jessica DeVicnezo, NYU-MRLJessica DeVicnezo, NYU-MRL

Robb Bifano, NYU-MRLRobb Bifano, NYU-MRL

Gloria Sed, NYU-ITP & Fashion DesignGloria Sed, NYU-ITP & Fashion Design

Chris Bregler, NYU-MRLChris Bregler, NYU-MRL

Sally RosenthalSally Rosenthal

Katie Salen, Parsons School of DesignKatie Salen, Parsons School of Design

Luke Dubois, Cycling74 / NYU-ITPLuke Dubois, Cycling74 / NYU-ITP

Tom Igoe, NYU-ITPTom Igoe, NYU-ITP

Michael Naimark, USC / NYU-ITPMichael Naimark, USC / NYU-ITP

Jon Meyer, NYU-MRLJon Meyer, NYU-MRL

Jeremi Sudol, NYU-MRLJeremi Sudol, NYU-MRL

Akex Postelnicu, RPIAkex Postelnicu, RPI

Michael Rabinovich, RPIMichael Rabinovich, RPI

Kevin Feeley, NYU-MRLKevin Feeley, NYU-MRL

Clilly Castiglia, NYU-MRLClilly Castiglia, NYU-MRL

Bo Wright, NYU-MRLBo Wright, NYU-MRL

Jessica DeVicnezo, NYU-MRLJessica DeVicnezo, NYU-MRL

Robb Bifano, NYU-MRLRobb Bifano, NYU-MRL

Gloria Sed, NYU-ITP & Fashion DesignGloria Sed, NYU-ITP & Fashion Design

Chris Bregler, NYU-MRLChris Bregler, NYU-MRL

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

From Single Pixels to Thousands of People From Single Pixels to Thousands of People

movement.nyu.edu squidball.netmovement.nyu.edu squidball.net