&RPSUHVVLRQ��3URFHVVLQJ��,QGH[LQJ�DQG�5HWULHYDO�RI��'�2EMHFWV�DQG�'DWD�

+RZ�WR�H[WHQG�LPDJH�YLGHR�SURFHVVLQJ�WR�JUDSKLFV"

Tsuhan ChenCarnegie Mellon University

tsuhan@cmu.edu

Joint work with Howard Leung, Masa Okuda, and Cha Zhang

7VXKDQ�&KHQ

�0LV�8QGHUVWDQGLQJ7R�JUDSKLFV�DQG�YLVLRQ�FRPPXQLWLHV9LGHR�LV�MXVW�ORZ�OHYHO SURFHVVLQJ

7R�WKH�YLGHR�FRPPXQLW\*UDSKLFV�LV�MXVW�VRPH�IDQF\�WRROV9LVLRQ�LV�WKLQJV�WKDW�GRQ¶W�ZRUN�LQ�SUDFWLFH

7VXKDQ�&KHQ

)LUVW�$WWHPSW«03(*��6WDUWHG�RXW�DV�PRGHO�EDVHG�FRGLQJ$QDO\VLV�DQG�V\QWKHVLV8VLQJ�YLVLRQ�JUDSKLFV�IRU�YLGHR�FRGLQJ

7KDW�GLGQ¶W�KDSSHQ��QRW�FRPSOHWHO\�6HWWOHG�ZLWK��'�VKDSH�EDVHG�FRGLQJ0RGHO�EDVHG�FRGLQJ�IRU�OLPLWHG�FRQWHQW��H�J���IDFHV

0RGHOLQJ�DQG�&RGLQJEXAMPLESMODELS CODED INFORMATION

PCM

Predictive CodingTransform Coding

Block-based codingH.261/263, MPEG-1/2

Model-based coding

MPEG-4

Pixels

Statistically dependent pixels

Moving blocks

Moving objects

Facial models

Moving regions

Color of pixels

Prediction error ortransform coeffs

Motion vectors and prediction error

Shapes, motion, and colors of objects

Action units

Shapes, motion, and colors of regions

Region-based codingH.263+, MPEG-4

MPEG-7A/V objects Description

7VXKDQ�&KHQ

0RGHOLQJ�DQG�&RGLQJ��FRQW��%HWWHU�PRGHOLQJ�LPSOLHV+LJKHU�FRPSUHVVLRQ0RUH�FRQWHQW�DFFHVVLELOLW\0RUH�FRPSOH[LW\/HVV�HUURU�UHVLOLHQFH

9LGHR�DQG�YLVLRQ�JUDSKLFV�GR�JR�KDQG�LQ�KDQG�DOO�DORQJ9LGHR�UHVHDUFK�LV�HYROXWLRQ�RI�YLVLRQ�DQG�JUDSKLFV�WHFKQLTXHV

7VXKDQ�&KHQ

7RSLFV&RPSUHVVLRQ�IRU�LPDJH�EDVHG�UHQGHULQJ

&RPSUHVVLRQ�IRU��'�PHVKHV6WUHDPLQJ�LQ�WH[WXUH�DQG�JHRPHWU\�MRLQWO\�

,QGH[LQJ�DQG�UHWULHYDO�RI��'�REMHFWV

%XLOGLQJ�LPPHUVLYH�HQYLURQPHQWV

7VXKDQ�&KHQ

,PDJH�%DVHG�5HQGHULQJ

««

>6KXP HW��DO@

7VXKDQ�&KHQ

&RPSUHVVLRQ

7KH�QXPEHU�RI�LPDJHV�LV�ODUJH��VR�ZH�QHHG�FRPSUHVVLRQ

*RRG�WR�KDYH�IHZHU�VDPSOHV'RHV�QRW�JXDUDQWHH�IHZHU�ELWV

&RQVLGHU�WKHVH�DV�D�YLGHR�VHTXHQFH*HQHUDO�YLGHR�FRGLQJ�DSSOLHV

7VXKDQ�&KHQ

DCT Q

IDCT

IQ

D

ME

MCMV

+

−�

�

IDCT

IQ

�D MC

MV

Network or Storage

Encoder Decoder

9LGHR�&RGHF

� ,QWUD�FRGLQJ� ,QWHU�FRGLQJ

�

� �

�

� �

Previous frame(reference frame)

Current frame

7VXKDQ�&KHQ

,QWUD�&RGLQJ

'LVDGYDQWDJH��'RHV�QRW�H[SORLW�WKH�FRUUHODWLRQ�EHWZHHQ�LPDJHV

,�IUDPH ,�IUDPH

««

� � �,�IUDPH

7VXKDQ�&KHQ

,QWHU�&RGLQJ

'LVDGYDQWDJH��'RHV�QRW�SURYLGH�UDQGRP�DFFHVV

L�H���IUDPH�1�GHSHQGV�RQ�IUDPH�1��

,�IUDPH 3�IUDPH

««

� � �3�IUDPH

7VXKDQ�&KHQ

3UHGLFWLRQ�IURP�6SULWH

��

««««

� � N�

>FI� $QDQGDQ HW��DO@

7VXKDQ�&KHQ

*HQHUDWLRQ�RI�6SULWHImage 1

Image 2

Image N-1

Image N

Image 1

Image 1

Image 2

Image N-1

Image N

Sprite

6WHS����)LQGLQJ�WKH�RIIVHW 6WHS����*HQHUDWLQJ�WKH�VSULWH

7VXKDQ�&KHQ

:HLJKWLQJ� QHHG�WR�ILQG�D�ZHLJKWLQJ�IXQFWLRQ�WR�EOHQG�WKH�

LPDJHV�WR�IRUP�WKH�VSULWH

0

0.001

0.002

0.003

0.004

0.005

0.006

0.007

0.008

0.009

0.01

0 100 200 300 400

Column number

Wei

ght

0

0.001

0.002

0.003

0.004

0.005

0.006

0.007

0.008

0.009

0.01

0 100 200 300 400

Column number

Wei

gh

t

0

0.02

0.04

0.06

0.08

0.1

0 100 200 300 400

Column number

Wei

gh

t

&RQVWDQW�ZHLJKWLQJ 7ULDQJXODU�ZHLJKWLQJ 'HOWD�ZHLJKWLQJ

7VXKDQ�&KHQ

&RQVWDQW�:HLJKWLQJ

««««

� � N�

7VXKDQ�&KHQ

7ULDQJXODU�:HLJKWLQJ

««««

� � N�

7VXKDQ�&KHQ

'HOWD�:HLJKWLQJ

««««

� � N�

7VXKDQ�&KHQ

0RGLILHG�&RGHF

� 3UHGLFWLRQ�IURP�VSULWH�LPDJH�ZLWKRXW�0&

DCT Q

Prediction

IQ+

−� IDCT

Network or Storage

Encoder Decoder

sprite + offset

Prediction

�

sprite + offset

7VXKDQ�&KHQ

:LWK�0RWLRQ�&RPSHQVDWLRQ

� 3UHGLFWLRQ�IURP�VSULWH�LPDJH�ZLWK�0&

DCT Q

Prediction

IQ

ME

MC

MV

+

−� IDCT

Network or Storage

Encoder Decoder

Sprite + offset

Prediction

MC

MV

Sprite + offset

�

7VXKDQ�&KHQ

:LWK�YV��:LWKRXW�0&

� �ZLWKRXW�0& ZLWK�0&

7VXKDQ�&KHQ

7HVW�6HTXHQFHV����

6\QWKHWLF�VHTXHQFH����1HW,&(�URRP 6\QWKHWLF�VHTXHQFH����3DUN

7VXKDQ�&KHQ

7HVW�6HTXHQFHV����

5HDO�VHTXHQFH����.LGV 5HDO�VHTXHQFH����.RQJPLDR

>6KXP��HW�DO@

7VXKDQ�&KHQ

:HLJKWLQJ�IXQFWLRQ�5HVXOWV

6\QWKHWLF�VHTXHQFH�� 6\QWKHWLF�VHTXHQFH��

5HDO�VHTXHQFH�� 5HDO�VHTXHQFH��

25

27

29

31

33

35

37

0 0.05 0.1 0.15 0.2

Bit rate (bpp)

PSN

R (d

B)

25

27

29

31

33

35

37

0 0.05 0.1 0.15 0.2

Bit rate (bpp)

PSN

R (d

B)

29

31

33

35

37

39

41

43

0 0.02 0.04 0.06 0.08 0.1

Bit rate (bpp)

PSN

R (d

B)

34

36

38

40

42

44

0 0.01 0.02 0.03 0.04 0.05

Bit rate (bpp)

PSN

R (d

B)

&RQVWDQW�ZHLJKWLQJ�������� 7ULDQJXODU�ZHLJKWLQJ���� 'HOWD�ZHLJKWLQJ

7VXKDQ�&KHQ

&RPSUHVVLRQ�5HVXOW

6\QWKHWLF�VHTXHQFH�� 6\QWKHWLF�VHTXHQFH��

29

31

33

35

37

39

41

0 0.05

Bit rate (bpp)

PS

NR

(dB

)

Intra coding Inter coding Mosais without MC Mosaic with MC

34

36

38

40

42

44

0 0.01 0.02 0.03 0.04 0.05

Bit rate (bpp)P

SN

R (d

B)

Intra coding Inter coding Mosais without MC Mosaic with MC

,QWUD�FRGLQJ���� ,QWHU�FRGLQJ���������6SULWH�ZLWKRXW�0&��������6SULWH�ZLWK�0&

7VXKDQ�&KHQ

&RPSUHVVLRQ�5HVXOW

5HDO�VHTXHQFH�� 5HDO�VHTXHQFH��

2526272829303132333435

0 0.05 0.1

Bit rate (bpp)

PS

NR

(dB

)

Intra coding Inter coding Mosais without MC Mosaic with MC

25262728293031323334

0 0.05 0.1

Bit rate (bpp)P

SN

R (d

B)

Intra coding Inter coding Mosais without MC Mosaic with MC

,QWUD�FRGLQJ���� ,QWHU�FRGLQJ���������6SULWH�ZLWKRXW�0&��������6SULWH�ZLWK�0&

7VXKDQ�&KHQ

(QKDQFHPHQWV:LQGRZ�VL]H�IRU�VHDUFKLQJ�RIIVHWV

6WULSH�PRWLRQ�FRPSHQVDWLRQ

0&�XVLQJ�D�ODUJH�UHIHUHQFH�IUDPH

0XOWLSOH�VSULWHVKids

26

27

28

29

30

31

32

33

0.2 0.3 0.4 0.5 0.6 0.7 0.8

bit rate (bpp)

PS

NR

(d

B)

1 sprite 3 sprites 5 sprites 7 sprites 9 sprites

7VXKDQ�&KHQ

5HFDS«6SULWH�SUHGLFWLRQ�ZLWK�0&�EHWWHU�WKDQ�,QWUD�FRGLQJ6SULWH�SUHGLFWLRQ�ZLWK�0&�LV�SUHIHUUHG�IRU�UDQGRP�DFFHVV%HWWHU�WKDQ�,QWHU�FRGLQJ�IRU�UHDO�GDWD

'HOWD�ZHLJKWLQJ�LV�WKH�EHVW�IRU�FRQVWUXFWLQJ�WKH�VSULWH&DQ�EH�H[WHQGHG�WR�KLJKHU�GLPHQVLRQV/XPLJUDSK��OLJKWILHOG��HWF�

7VXKDQ�&KHQ

6WUHDPLQJ��'

��VHFRQG

���VHFRQGV���VHFRQGV

*HRPHWU\�7H[WXUH

7VXKDQ�&KHQ

7H[WXUH���*HRPHWU\� ��'�2EMHFW

&RUQHU�%DVHG

9HUWH[�%DVHG

7VXKDQ�&KHQ

:K\�&RPSUHVVLRQ"(DFK�YHUWH[��WKUHH�IORDWLQJ�SRLQW�QXPEHUV,I�HDFK�YHUWH[�VKDUHG�E\���WULDQJOHV��DQG�PD[�QXPEHU�RI�YHUWLFHV�SHU�PRGHO�LV����

⇒ ����ELWV�WULDQJOH�QHHGHG

⇒ ���.%a�0%�IRU�DQ�DYHUDJH�PRGHO���WH[WXUH

triangle

bits

IDvertex

bits

triangle

IDsvertex

vertex

bits

triangle

vertices 10820*

33*32*

3*

6

1 =

+

&RPSUHVVLRQ�RI��'�2EMHFWV7H[WXUH�FRPSUHVVLRQ6WDWLF�WH[WXUHV��-3(*�RU�-3(*�����'\QDPLF�WH[WXUHV��03(*�RU�+����

*HRPHWU\�FRPSUHVVLRQ4XDQWL]DWLRQ�RI�YHUWH[�FRRUGLQDWHV3UHGLFWLYH�FRGLQJ(QWURS\�FRGLQJ

*UDQXODU�VWDEOH�SURJUHVVLYH�FRGLQJ0HVK�RSWLPL]DWLRQ�VLPSOLILFDWLRQ�

>+RSSH�HW�DO@>+HFNEHUW HW�DO@>6FKURGHU HW�DO@>7DXELQ HW�DO�@

7VXKDQ�&KHQ

7H[WXUH�&RGLQJ

%ORFN�'LDJUDP

9HUWH[�4XDQWL]DWLRQ

(QWURS\�&RGLQJ

9HUWH[�&RRUGLQDWHV

3UHGLFWLRQ

%LWVWUHDP

&RQQHFWLYLW\��

�

�'�0RGHO7H[WXUH

7VXKDQ�&KHQ

(QFRGLQJ9HUWH[�GHFLPDWLRQ

C

C

1

234

56

165

4

5H�WULDQJXODWLRQ

( )iv

7VXKDQ�&KHQ

,PSRUWDQFH�RI�9HUWLFHV

��� 9ROXPH

��� &RORU

)(iv

)(ic

V1V2

7VXKDQ�&KHQ

5DQN�DOO�YHUWLFHV�IURP�KLJK�WR�ORZ�EDVHG�RQ�D�FRVW�IXQFWLRQ�

Y��L���LV�WKH�JHRPHWU\�FRVWF��L���LV�WKH�WH[WXUH�FRORU�QRUPDO�FRVWα LV�DQ�XVHU�VSHFLILHG�SDUDPHWHU

'HFLPDWH�WKH�YHUWLFHV�ZLWK�ORZ�FRVW�ILUVW7UDQVPLW�WKH�YHUWLFHV�ZLWK�KLJK�FRVW�ILUVW

)()1()()( icivim αα −+=

7VXKDQ�&KHQ

&RGLQJ�RI�7H[WXUH9HUWH[�EDVHG:DYHOHW��63,+7����HQWURS\�FRGLQJ

&RUQHU�EDVHG3DGGLQJ���'&7���UXQ�OHQJWK�FRGLQJ���HQWURS\�FRGLQJ7H[WXUH�UH�PDSSLQJ�QHHGHG

7VXKDQ�&KHQ

7H[WXUH�5H�0DSSLQJ

m

v

vm

7VXKDQ�&KHQ

���

���

���

��

���

���

��

��

950/

��J]LS

WH[WXUH����9DVH

WH[WXUH���'XFN

QRQH����3LHWD

WH[WXUH����7RWHP

QRQH����+RUVH

QRQH����&URFRGLOH

1RQH����&RZ

1RQH����%HHWKRYHQ

$WWULEXWHV03(*��2XU

$OJRULWKP

�LQ�.%\WHV�

&RPSDULVRQ

7VXKDQ�&KHQ

9LHZ�$GDSWLYH�7UDQVPLVVLRQ

Viewpoint B

Viewpoint A

Hypothetical Viewpoint

7VXKDQ�&KHQ

5HWULHYDO�RI��'�2EMHFWV,QGH[LQJ�DQG�UHWULHYDO0XFK�LV�GRQH�IRU�LPDJHV>+XDQJ�HW�DO@>&R[�HW�DO@

5HFHQW�ZRUN�IRU��'�REMHFWV5HODWHG�WR�03(*��

)HDWXUH�H[WUDFWLRQ

)HDWXUH�PDWFKLQJ

7VXKDQ�&KHQ

)HDWXUH�([WUDFWLRQ)HDWXUH�H[WUDFWLRQ7UDGLWLRQDOO\�YHUWH[�VXUIDFH�EDVHG1HZ�UHJLRQ�EDVHG�IHDWXUHVPRPHQW�LQYDULDQWV��)RXULHU�WUDQVIRUP�FRHIILFLHQWV��HWF��

3UHSURFHVVLQJ�WR�FORVH�WKH�PRGHO

6XUIDFH 5HJLRQ

7VXKDQ�&KHQ

)HDWXUH�([WUDFWLRQ��FRQW��(IILFLHQWO\�FDOFXODWH�UHJLRQ�EDVHG�IHDWXUH�GLUHFWO\�IURP�PHVK6LJQHG�IHDWXUH�IRU�HDFK�PHVK�HOHPHQW5REXVW�WR�WULDQJXODWLRQ$SSOLHV�WR�DQ\�IHDWXUH�WKDW�FDQ�EH�GHFRPSRVHG�WR�HDFK�PHVK�HOHPHQW +

−−

−+

+

7VXKDQ�&KHQ

�'�0RGHO�5HWULHYDO

7VXKDQ�&KHQ

$QQRWDWLRQ�DQG�$FWLYH�/HDUQLQJ

6HPDQWLF�WKUX�DQQRWDWLRQ�LV�QHHGHG/RZ�OHYHO�IHDWXUHV�QRW�HQRXJK+LHUDUFKLFDO�DQQRWDWLRQ&RPSDWLEOH�FRQFHSWV�LQ�DQQRWDWLRQ

$FWLYH�OHDUQLQJ&RPSOHWH�DQQRWDWLRQ�LV�LPSUDFWLFDO6HOHFW�WKH�REMHFW�PRVW�XQFHUWDLQ�IRUDQQQRDWLRQ

7VXKDQ�&KHQ

$QQRWDWLRQ

7VXKDQ�&KHQ

$FWLYH�/HDUQLQJ)RU�HDFK�PRGHO��HDFK�FRQFHSW��ZH�PDLQWDLQ�D�SUREDELOLW\�RI�WKLV�PRGHO�EHORQJLQJ�WR�WKLV�FRQFHSW

6HW�WKH�SUREDELOLW\�WR���RU���LI�DQQRWDWHG

(VWLPDWH�SUREDELOLWLHV�RI�WKH�XQODEHOHG�REMHFWV�ZLWK�SRWHQWLDO�IXQFWLRQ

8VH�WKH�SUREDELOLWLHV�WR�HVWLPDWH�XQFHUWDLQW\�DQG�WR�PHDVXUH�WKH�VHPDQWLF�GLVWDQFH

7VXKDQ�&KHQ

$FWLYH�/HDUQLQJ

-5 -4 -3 -2 -1 0 1 2 3 4 50.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

max2 d⋅

d

p )/exp(5.05.0 2max

20 ddcp ⋅−+=

-10 -8 -6 -4 -2 0 2 4 6 8 100.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

f

kp

Annotated models

One annotated neighborhood Multiple annotated neighborhoods

The potential function

7VXKDQ�&KHQ

(VWLPDWH�WKH�8QFHUWDLQW\

1

p2p

iep

7VXKDQ�&KHQ

5HVXOWV

0

0.5

1

1.5

2

2.5

3

0 50 100 150 200 250 300 350

# of Samples Annotated

Ret

riev

al P

erfo

rman

ce (D

)

Best Gradient Search

Random Sampling

Our Algorithm

0

1

2

3

4

5

6

0 50 100 150 200 250

# of Models Annotated

Ret

riev

al P

erfo

rman

ce (D

) Best Gradient Search

Random Sampling

Our Algorithm

Synthetic database A small database

7VXKDQ�&KHQ

5HVXOWV��FRQW��

0

1

2

3

4

5

6

7

8

9

0 500 1000 1500 2000

# of models annotated

Ret

riev

al P

erfo

rman

ce (

D) Random Sampling

Our algorithm

7VXKDQ�&KHQ

5HFDS«1HZ�IHDWXUH�VHW�IRU��'�PRGHOV

$FWLYH�OHDUQLQJ�WR�LPSURYH�DQQRWDWLRQ�HIILFLHQF\

&RPSDWLEOH�FRQFHSW�WUHH�IRU�DQQRWDWLRQ

3UREDELOLW\�IRU�ERWK�XQFHUWDLQW\�HVWLPDWLRQ�DQG�VHPDQWLF�GLVWDQFH

“Collaboration from anywhere, through any media, as if face-to-face in one room”

Network

,PPHUVLYH�(QYLURQPHQWV

7VXKDQ�&KHQ

7VXKDQ�&KHQ

$�3URWRW\SH1HW,&(��1HWZRUNHG�,QWHOOLJHQW�&ROODERUDWLYH�(QYLURQPHQW/LS�V\QF�IDFLOLWDWHV�VSHHFK�XQGHUVWDQGLQJ

:KR�LV�VSHDNLQJ�DQG�ZKDW�LV�EHLQJ�VDLG

&RQVLVWHQW�VSDWLDO�UHODWLRQVKLS�ZLWK�H\H�FRQWDFW:KRP�LV�VSRNHQ�WR

)DFLDO�H[SUHVVLRQV�DQG�YRLFH�GULYHQ�KDQG�JHVWXUHV'LUHFWLRQDO�VRXQG�JLYH�VHQVH�RI�GLVWDQFH�DQG�GLUHFWLRQ

:KR�LV�ZKHUH��:KR�LV�VSHDNLQJ(QDEOH�VPDOO�JURXS�LQWHUDFWLRQ�LQ�D�URRP�IXOO�RI�SHRSOH

,QIRUPDWLRQ�VKDULQJ6KDUHG�ZKLWHERDUG6WUHDPLQJ��'�REMHFWV(QDEOH�FROODERUDWLYH�GHVLJQ��H�J���FDUV��EXLOGLQJV��HWF�

7VXKDQ�&KHQ

1HW,&(

7VXKDQ�&KHQ

1HW,&(

7VXKDQ�&KHQ

1HW,&(

7VXKDQ�&KHQ

&DVH�6WXG\��2QOLQH�$XFWLRQ

7VXKDQ�&KHQ

2QJRLQJ�:RUN8VH�,%5�IRU�EDFNJURXQG�UHQGHULQJ

8VHU�VWXG\7RJHWKHU�RU�RQ�ORFDWLRQ

7UDFNLQJ�IRU�UHQGHULQJ+HDG�WUDFNLQJ�IRU�KHDG�RULHQWDWLRQ*D]H�WUDFNLQJ�IRU�H\H�FRQWDFW+DQG�WUDFNLQJ�IRU�KDQG�JHVWXUHV

7VXKDQ�&KHQ

6XPPDU\&RPSUHVVLRQ�IRU�,%5

&RPSUHVVLRQ�IRU��'�PHVKHV

,QGH[LQJ�DQG�UHWULHYDO�RI��'�REMHFWV

,PPHUVLYH�HQYLURQPHQWV

7VXKDQ�&KHQ

$GYDQFHG�0XOWLPHGLD�3URFHVVLQJ�/DE

3OHDVH�YLVLW�XV�DW�

KWWS���DPS�HFH�FPX�HGX