7/29/2019 7-delayedfeedback

1/19

,1)/8(1&(2)'(/$

7/29/2019 7-delayedfeedback

2/19

SHUIRUPDQFHLVVWLOOGLVSXWDEOH,QWHUPVRIVWUDWHJ\VHDUFKLWLVJHQHUDOO\DFFHSWHGWKDWWKHSRVLWLYHIHHGEDFNUHLQIRUFHVPDLQWDLQLQJDQGLPSURYLQJFXUUHQWVWUDWHJ\DQGQHJDWLYHIHHGEDFNXUJHVWRVHDUFKIRUQHZVWUDWHJLHV6FKZDUW]>@+RJDUWKHWDO>@EXWOLWWOHZRUN KDV EHHQ GRQH RQ LQYHVWLJDWLQJ WKH LPSDFW RI GHOD\HG IHHGEDFN RQ WKH VWUDWHJ\VHDUFK,QWHUPVRIWLPHFRQVXPSWLRQRIGHFLVLRQPDNLQJ6FKRROHUDQG$QGHUVRQ>@IRU H[DPSOH UHSRUW WKDW VWXGHQWV OHDUQLQJ /,63 UHFHLYLQJ LPPHGLDWH IHHGEDFN PRYHGWKURXJKWKHWUDLQLQJPDWHULDOLQOHVVWLPHWKDQWKRVHUHFHLYLQJGHOD\HGIHHGEDFN\HWZLWKRXWGHWULPHQWDOHIIHFWVRQOHDUQLQJ,QWKHVHFRQGH[SHULPHQWGXULQJWKHVDPHVWXG\XVLQJLPSURYHG/,63HGLWRUVXEMHFWVUHFHLYLQJLPPHGLDWHIHHGEDFNZHUHVWLOOIDVWHUEXW WKH\ZHUH VORZHURQ WKH WHVWSUREOHPVDQGFRPPLWWHG WZLFHDVPDQ\HUURUV DVWKHVWXGHQWV UHFHLYLQJ GHOD\HG IHHGEDFN )LQDOO\ LQ WHUPV RI WKH LQIOXHQFH RI WLPLQJ RIIHHGEDFNRQHPRWLRQDOVWDWHVOHWPHMXVWFLWHDIWHU-HQVHQ

VWXGLHVUHYHDOWKDWIRUKLJK

ULVN RU IRU KLJKO\ VWUHVVHG VWXGHQWV WKH PRVW EHQHILFLDO IRUP RI IHHGEDFN LV GHOD\HGIHHGEDFN,WKHOSVPHGLDWHLPSXOVLYLW\

7KHSXUSRVHRIWKLVSDSHULVWRUHYLVLWWKHLQIOXHQFHRIWKHGHOD\HGIHHGEDFNRQ

OHDUQLQJSHUIRUPDQFHDQGVWUDWHJ\VHDUFK7KHH[SHULPHQWFRQGXFWHGE\+RJDUWKHWDO>@LQYHVWLJDWLQJWKHLQIOXHQFHRIH[DFWLQJQHVVRIHQYLURQPHQWVDQGILQDQFLDOLQFHQWLYHVRQOHDUQLQJDQGSHUIRUPDQFHKDVEHHQDGDSWHGLQRUGHUWRVWXG\WKHLQIOXHQFHRIGHOD\HGIHHGEDFN RQ OHDUQLQJ DQG SHUIRUPDQFH XQGHU HQYLURQPHQWV SHQDOL]LQJ WKH HUURUV RIGHFLVLRQPDNHUVZLWKGLIIHUHQWVHYHULW\7ZRPDLQZRUNLQJK\SRWKHVHVKDYHEHHQXVHGDVDJXLGHOLQHRIWKHVWXG\QDPHO\WKDWZKHQOHDUQLQJHQYLURQPHQWLVOHQLHQWHUURUVDUHQRW KHDYLO\ SXQLVKHG DV D FRQVHTXHQFH RI ZKLFK JHQHUDOO\ SRVLWLYH IHHGEDFN LVREVHUYHGSRVWSRQHGIHHGEDFNDLGVOHDUQLQJSURFHVVWRWKHGHWULPHQWRIWKHSHUIRUPDQFHDWOHDVWLQWKHVKRUWUXQ:KHQOHDUQLQJHQYLURQPHQWLVH[DFWLQJSHQDOWLHVLPSRVHG

IRU HUURUV DUH VHYHUH WKHUHIRUH JHQHUDOO\ QHJDWLYH IHHGEDFN LV REVHUYHG SRVWSRQHGIHHGEDFN DGYHUVHO\DIIHFWV OHDUQLQJSURFHVV DQG SHUIRUPDQFH:KLOH WKH GDWD IURP WKHH[SHULPHQWGLVFRQILUPHGERWKWKHVHVLWVKRZHGDQLQWHUHVWLQJSKHQRPHQRQZKLFKKDVQRWEHHQFRQMHFWXUHGEHIRUHQDPHO\WKDWWKHWLPLQJRIIHHGEDFNFDQLQIOXHQFHVXEMHFWVTXHVWIRUVWUDWHJ\RISUREOHPVROYLQJGHSHQGLQJRQWKHH[DFWLQJQHVVRIWKHHQYLURQPHQW 7KHUHVWRIWKLVSDSHULVRUJDQL]HGDVIROORZV)LUVW,GLVFXVVWKHLPSRUWDQFHRIVWXG\LQJWKHWLPLQJRIIHHGEDFNDQGWKHFRQWURYHUV\DURXVHGDURXQGWKLVLVVXH6HFRQG,UHSRUW WKH UHVXOWV RIWKHH[SHULPHQW LQYHVWLJDWLQJ WKH LQIOXHQFHRI GHOD\HG IHHGEDFNRQOHDUQLQJSHUIRUPDQFHDQGVWUDWHJ\VHDUFK(YHQWXDOO\,FRQFOXGHWKHZKROHZLWKDILQDOGLVFXVVLRQSRVVLEOHFULWLFLVPDQGVRPHH[DPSOHVRIDSSOLFDWLRQVRIP\ILQGLQJV

'(/$S@

7KHLQWHUQDWLRQDO$VVRFLDWLRQRI6XSHUYLVLRQDQG&XUULFXOXP'HYHORSPHQW'RZQORDGHGRQ-XQHIURPKWWSZZZFDVFGRUJVSRWMHQVHQVKWPO

7/29/2019 7-delayedfeedback

3/19

7KH UHVHDUFK RQ IHHGEDFN GDWHV EDFN WR WKH HDUO\ GD\V RI SV\FKRORJ\ IRU

H[FHOOHQWUHYLHZRIIHHGEDFNVWXGLHVFKHFN0RU\>@RU%ULQNR>@DQGIURPWKHYHU\HDUO\ RQ IHHGEDFN KDV EHHQ FRQVLGHUHG D PDLQ FRQWULEXWLQJ IDFWRU WR NQRZOHGJHDFTXLVLWLRQ 2QH RI WKH PRVW LPSRUWDQW SULQFLSOHV LQ DOO OHDUQLQJ WKHRU\ ODZ RIUHLQIRUFHPHQWRUODZRIHIIHFWODLGDVFLHQWLILFEDVHIRUIXWXUHIHHGEDFNVWXGLHV,QWKLVUXOHIRUVKDSLQJEHKDYLRUE\WKHXVHRIUHZDUGVUHLQIRUFHUV6NLQQHU+LOJDUGDQG%RZHU>S@ DIWHU6NLQQHUGHILQHVSULPDU\SRVLWLYHQHJDWLYH UHLQIRUFHU DVD VWLPXOXVZKLFK ZKHQ DSSOLHG UHPRYHG IROORZLQJ DQ RSHUDQW UHVSRQVH VWUHQJWKHQV WKHSUREDELOLW\RI WKDW UHVSRQVH+HQDPHV IRRGZDWHUDQG VH[XDOFRQWDFW DVH[DPSOHV RISRVLWLYHUHLQIRUFHUVDQGYHU\EULJKWOLJKWORXGQRLVHRUH[WUHPHKHDWRUFROGDVH[DPSOHVRIQHJDWLYHUHLQIRUFHUV+LOJDUG+LOJDUGDQG%RZHU>S@DIWHU+LOJDUGPHQWLRQVDOVR OHDUQHG VHFRQGDU\ UHLQIRUFHUV OLNH PRQH\ SUDLVH VRFLDO DSSURYDO DWWHQWLRQGRPLQDQFHHWF

,WLVJHQHUDOO\DJUHHGWKDWDOHDUQDEOHUHVSRQVHIROORZHGE\DUHLQIRUFLQJHYHQWVWLPXOXVVWDWHRIDIIDLUVHWFZLOOUHFHLYHDQLQFUHPHQWLQLWVVWUHQJWKRUSUREDELOLW\RIRFFXUUHQFH%XWIHHGEDFNDVLWLVGHILQHGLQWKLVSDSHUPLJKWGLIIHUIURPDUHLQIRUFHU$VUHLQIRUFHUVXVHGIRUFRQGLWLRQLQJDUHXVXDOO\ELQDU\\HVQRUHZDUGSXQLVKPHQWHWFIHHGEDFNLQDGHFLVLRQPDNLQJ WDVN FDQ DGGLWLRQDOO\ FRQYH\ DQ H[WHQGHG HYDOXDWLYH FRPSRQHQWPDNLQJXVHRIWKHZKROHVFDOHEHWZHHQH[WUHPHYDOXHV+RZHYHUWKHYLUWXHRIIHHGEDFNLQ GHFLVLRQPDNLQJ FRQYHUWV LQ LWV PDLQ YLFH 7KH QDWXUDO DPELJXLW\ RI IHHGEDFNFRQWLQXDOO\FRQIRXQGVWKHLQIHUHQWLDODQGWKHHYDOXDWLYHUROHRIWKHLQIRUPDWLRQEHLQJIHGEDFNWRWKHGHFLVLRQPDNHU/HWXVXVHDQH[DPSOHIURP+RJDUWKHWDO>@RIDVWXGHQWZULWLQJDILQDOSDSHUIRUKLVDFDGHPLFFRXUVH$IWHUUHFHLYLQJKHUJUDGHWKHVWXGHQWFDQHDVLO\JHWFRQIRXQGHGZLWK KRZPXFKRIWKHJUDGHVKRXOGVKHDWWULEXWHWRKHUSDSHU

ZULWLQJVNLOOVDQGKRZPXFKWRWKHHYDOXDWLRQSROLF\RIKHUWHDFKHU,WLVWKHQQRWTXHVWLRQDEOHWKDWLQDOPRVWDOOKXPDQOHDUQLQJVLWXDWLRQVWKHUROHRI

IHHGEDFN LV WR FRQYH\ LQIRUPDWLRQ DERXW WKH FRUUHFWQHVV RI JLYHQ UHVSRQVH 7KLVIHHGEDFNPRVWSUREDEO\ KDVLWV URRWV LQORQJ KLVWRU\RIFXOWXUDO WUDLQLQJZKHQ EHLQJFRUUHFWLVDVVRFLDWHGZLWKSDUHQWDOSUDLVHDQGDSSURYDO:KDWHYHURXURSLQLRQDERXWWKHSURFHVVRIOHDUQLQJLVZKHWKHUZHWHQGWRV\PSDWKL]HZLWK OHDUQLQJE\GRLQJDSSURDFKRUZLWKVRFLDOOHDUQLQJDSSURDFKDODUJHDPRXQWRIKXPDQOHDUQLQJLVGRQHYLFDULRXVO\WKURXJK REVHUYLQJ DQRWKHU SHUVRQ PDNLQJ WKH VNLOOHG UHVSRQVHV DQG WKHQ WU\LQJ WRLPLWDWHWKHLPSRUWDQFHRIIHHGEDFNLVXQGHQLDEOH

2YHUWLPHGLIIHUHQWLQYHVWLJDWLRQVKDYHEHHQSHUIRUPHGDQGDORWKDVEHHQZULWWHQDERXWWKHDSSURSULDWHWLPLQJRIIHHGEDFN%XWVWLOOGDQJHURXVP\WKVFRH[LVWZLWKSURYHQ

IDFWV 2QH RI WKHP LV WKDW OHDUQLQJ HIILFLHQF\ LV DOZD\V LQFUHDVHG WKURXJK WKH XVH RILPPHGLDWHIHHGEDFN)RUWKHSXUSRVHRIP\DQDO\VLV,ZLOOGLYLGHOHDUQLQJVLWXDWLRQVDVLWLV FRPPRQ LQ OHDUQLQJ OLWHUDWXUH VHH &KDU >@ LQWR WZR EURDG JURXSV ORZHU RUGHUOHDUQLQJ WDVNV UHTXLULQJ NQRZOHGJH DQG LWV DSSOLFDWLRQ DQG KLJKHURUGHU OHDUQLQJWDVNVUHTXLULQJDQDO\VLVV\QWKHVLVDQGHYDOXDWLRQ)RUWKHIRUPHU6NLQQHULDQDSSURDFKWUHDWLQJWKHIHHGEDFNDVUHLQIRUFHPHQWVHHPHGODWHUWREHFRQILUPHGE\RWKHUUHVHDUFKHUVIRUH[DPSOH%DQJHUW'URZQVHWDO>@%HUTXLVWDQG3KLOLSV>@ZKRUHDFKHGDJHQHUDOFRQFOXVLRQ WKDW LQ W\SLFDO FODVVURRP VHWWLQJV WKH LPPHGLDWH IHHGEDFN KDV EHHQ PRUHHIIHFWLYH7KH VDPH KDVEHHQ IRXQG LQ RQOLQH OHDUQLQJ 2JLOYLH

$OVR +RJDUWK >@

2JLOYLHKWWSVZHEFWDLWLDVWDWHHGX,68WRROVZHEKWPOGHVLJQHUFRPPXQLW\RJLOYLHSGI

7/29/2019 7-delayedfeedback

4/19

EDVLQJRQ=DMRQFVDVVXPSWLRQWKDWLWLVPRUHLPSRUWDQWWROHDUQZKHQVRPHWKLQJQHZLVSRWHQWLDOO\ KDUPIXO RU GDQJHURXV WKDQ ZKHQ LW LV QRW SUHVHQWV D PDWUL[ RI OHDUQLQJVWUXFWXUHWDNLQJDVGLPHQVLRQVWKHFRQVHTXHQFHRIHUURUDQGWKHTXDOLW\RIIHHGEDFN+HFODLPV WKDW ZLWK VSHHG\ DQG DFFXUDWH IHHGEDFN ZH FDQ OHDUQ ERWK HDVLO\ DQG

DFFXUDWHO\+RZHYHUZKHQIHHGEDFNLVQRLV\GHOD\HGRUFKDUDFWHUL]HGE\XQFHUWDLQW\

WKHSURFHVVRIOHDUQLQJE\FRQQHFWLRQVDQGUHLQIRUFHPHQWFDQEUHDNGRZQLHZKDWZHOHDUQPD\QRWEHYDOLG

:KLOH SUDLVLQJ WKH VHDUFK IRU LPPHGLDWH DQG UHOLDEOH IHHGEDFN LV REYLRXVO\FRUUHFWDQGYDOXDEOH,ZRXOGOLNHWRFRQWUDVWLWZLWKVRPHRWKHUHYLGHQFH0RUULVRQHWDO>@ IRXQG WKDW GHOD\HG DQG NQRZOHGJHRIFRUUHFWUHVSRQVH IHHGEDFN PD\ EH PRUHEHQHILFLDO WKDQ DQVZHUXQWLOFRUUHFW RU QRIHHGEDFN IRU ORZHU OHYHO OHDUQLQJ EXW WKDWIHHGEDFNHIIHFWVEHFRPHZHDNHUZKHQKLJKHURUGHUXQGHUVWDQGLQJLVHYDOXDWHG6LPLODUUHVXOWV KDYH EHHQ REWDLQHG E\ &ODULDQD >@ IHHGEDFN HIIHFWV ZHDNHU IRU KLJKHU RUGHUOHDUQLQJ.XOKDY\>@UHSRUWVVWXGLHVVKRZLQJWKDWGHOD\LQJWKHIHHGEDFNIRUDGD\RU

PRUHUHVXOWVLQVLJQLILFDQWLQFUHDVHVLQVWXGHQWVNQRZOHGJHUHWHQWLRQRQSRVWWHVWVFRUHV'HOD\5HWHQWLRQ(IIHFW7KLVSKHQRPHQRQZDVREVHUYHGPRVWO\LQPXOWLSOHFKRLFHVWHVWLQJDQG LWVH[SODQDWLRQKDVEHHQDWWULEXWHG WRVR FDOOHGLQWHUIHUHQFHSHUVHYHUDWLRQK\SRWKHVLV E\ .XOKDY\ DQG $QGHUVRQ >@ 7KH DXWKRUV IRXQG RXW WKDW LI IHHGEDFN LVGHOD\HG WKH LQLWLDO HUURUV WHQG WR EH IRUJRWWHQ DQG GR QRW LQWHUIHUHZLWK OHDUQLQJ WKHFRUUHFWIHHGEDFNUHVSRQVHV

)RUWKHKLJKHURUGHUOHDUQLQJVLWXDWLRQVVXSHULRULW\RIWKHLPPHGLDWHIHHGEDFNRYHU WKH GHOD\HG IHHGEDFN LV DOVR QRW XQDQLPRXVO\ DJUHHG0DGGR[ HW DO >@ZKLOHLQYHVWLJDWLQJ WKH GHOD\HG IHHGEDFN HIIHFWV RQ UXOHEDVHG DQG LQIRUPDWLRQLQWHJUDWLRQFDWHJRU\OHDUQLQJVWDWH OHDUQLQJLQUXOHEDVHGWDVNVLVGRPLQDWHGE\DQH[SOLFLWV\VWHPWKDWXVHVZRUNLQJPHPRU\DQGH[HFXWLYHDWWHQWLRQ7KLVV\VWHPDSSHDUVWROHDUQWKURXJK

D FRQVFLRXV SURFHVV RI K\SRWKHVLV JHQHUDWLRQ DQG WHVWLQJ *LYHQ WKLV K\SRWKHVLVPDQLSXODWLRQVRIWKHWLPLQJDQGSODFHPHQWRIFRUUHFWLYHIHHGEDFNVKRXOGDQGGRKDYH

OLWWOHHIIHFWRQUXOHEDVHGOHDUQLQJ,QIDFWDVVXJJHVWHGE\$VKE\HWDOVUHVXOWVSHRSOHFDQOHDUQVRPHUXOHEDVHGFDWHJRULHVZLWKQRIHHGEDFNRIDQ\NLQG*D\QRU>@DQG5RSHU>@KDVLQGLFDWHGWKDWORZDFKLHYLQJVWXGHQWVPD\EHDEOHWREHQHILWPRUHIURP LPPHGLDWH IHHGEDFNZKHUHDV KLJK DFKLHYLQJ VWXGHQWV WHQG WRPDNH D EHWWHU XVHIURP GHOD\HG IHHGEDFN 7KLV IDFW ZDV FRQWULEXWHG WR WKH DELOLW\ RI KLJK SHUIRUPLQJVWXGHQWVWRDQDO\]HFXPXODWHGLQIRUPDWLRQDQGKDYLQJWLPHUHWKLQNWKHLQFRUUHFWRQH0DVRQ DQG %UXQLQJ >@ DIWHU WKRURXJKO\ DQDO\]LQJ WKH OLWHUDWXUH GHDOLQJ ZLWKIHHGEDFNVWLPLQJVWDWHWKDWIRUWKHLQVWUXFWLRQDLPLQJDWGHYHORSLQJKLJKHURUGHUVNLOOVVXFKDVFRPSUHKHQVLRQDSSOLFDWLRQRU DEVWUDFW UHDVRQLQJ WKHPRVWHIIHFWLYHNLQGRI

IHHGEDFNVHHPWREHWKHGHOD\HGRUHQGRIVHVVLRQRQH)LQDOO\)DUTXKDU>@FRQGXFWLQJH[SHULPHQWVRQGHFLVLRQPDNLQJLQ KLVPLFURZRUOG/2$'(5 ILQGVRXW WKDWZKHQ FRUUHFWLYH IHHGEDFN ZDV JLYHQ WKH GHOD\ RI IHHGEDFN IRU QRQFULWLFDO HUURUV

UHVXOWHGLQLPSURYHGSHUIRUPDQFHRYHUWKHXVHRILPPHGLDWHIHHGEDFNIRUDOOHUURUV,QWHUPVRILQIOXHQFHRIIHHGEDFNVWLPLQJRQVWUDWHJ\VHDUFKQRWPXFKZRUNKDV

EHHQ GRQH %XFKZDOG +LOJDUG DQG %RZHU > S @ DIWHU %XFKZDOG SURSRVHV KLVGHOD\HGLQIRUPDWLRQH[SHULPHQWEHLQJDUHVSRQVHWRWKH7KRUQGLNHDQ>@H[SODQDWLRQRIWKH VSUHDGRIHIIHFW7KRUQGLNHVHW XS WKH IROORZLQJH[SHULPHQWD OLVW RIQRXQVZDVSUHVHQWHGWRWKHVXEMHFWVZKRPXVWKDYHJXHVVHGDGLJLWIURPWRDVVRFLDWHGZLWKHDFKQRXQ7KHH[SHULPHQWHUNHSWVD\LQJDUELWUDU\ZURQJWRDOORIWKHDWWHPSWVH[FHSWOHW

7/29/2019 7-delayedfeedback

5/19

VD\WKHILIWKLWHP)ROORZLQJWKHILUVWF\FOHDVHFRQGLGHQWLFDOF\FOHZDVFRQGXFWHG,Q WKH LOOXVWUDWLRQ 7DEOH ZH FDQ VHH D UHSHWLWLRQ RI UHZDUGHG SDLU SHQFLO ,QWHUHVWLQJO\ZHDOVRVHHDQH[DFWUHSHWLWLRQRIUHVSRQVHVJLYHQLQWKHILUVWWULDORQWKH

UG

DQG WK SRVLWLRQ EHIRUH WKH UHZDUGHG SDLU DQG RQ WKH WKDQGWK SRVLWLRQ DIWHU WKHUHZDUGHGSDLU7KLVSKHQRPHQRQZDVFDOOHGWKHVSUHDGRIHIIHFW7DEOH,OOXVWUDWLYHHYHQWVRQWZRWULDOVRID7KRUQGLNHDQDQG%XFKZDOGVDVVRFLDWLYHOHDUQLQJWDVN7KRUQGLNH%XFKZDOG7ULDO7ULDO7ULDO6HULDO3RVLWLRQ&XH:RUG6V*XHVV(V)HHGEDFN6V*XHVV6V*XHVV

FXS ZURQJ

VFLVVRUV ZURQJ SODWH ZURQJ VN\ ZURQJ SHQFLO 5,*+7 ERRN ZURQJ KRXVH ZURQJ FKDLU ZURQJ

,QFRQWUDVW%XFKZDOGUXQWKHVDPHNLQGRIH[SHULPHQWZLWKRQO\GLIIHUHQFHRI

ZLWKKROGLQJ VXEMHFWV IHHGEDFN GXULQJ WKH ILUVW WULDODQGUHYHDOLQJ LWVKRUWO\ EHIRUH WKHVHFRQG WULDO GHFLVLRQVZHUH WR EH WDNHQ ,Q WKLV FDVH RQO\ WKH UHZDUGHG SDLU KDV EHHQUHSHDWHG 7KRUQGLNHDQ SUHGLFWLRQ WKDW VXEMHFWV UHFHLYLQJ LPPHGLDWH IHHGEDFN VKRXOGVKRZ ODUJHU WHQGHQF\ WR YDU\ WKHLU UHVSRQVHV KDSSHQHG WR EH IDOVH VLQFH LW ZDV%XFKZDOGV VXEMHFWV ZLWK GHOD\HG IHHGEDFN ZKR VKRZHGPRUH YDULDELOLW\ LQ DQVZHUV%XFKZDOGVH[SODQDWLRQRI WKHVSUHDGRIHIIHFW SKHQRPHQRQLV WKH IROORZLQJ:KLOH7KRUQGLNHDQVXEMHFWVKDGWRUHPHPEHUDOOFRPELQDWLRQVRI6WLPXOL5HVSRQVH2XWFRPHIURP WKH ILUVW WULDO6WLPXOL 5HVSRQVH5HVSRQVH 2XWFRPHDQG 6WLPXOL 2XWFRPH%XFKZDOGVVXEMHFWVKDGWRUHPHPEHURQO\WKH6WLPXOL5HVSRQVHDVVRFLDWLRQ

5HFHQWO\ WKH HPHUJHQFH RI LQIRUPDWLRQSURFHVVLQJ WKHRU\ RSHQHG QHZRSSRUWXQLWLHV IRU IHHGEDFNV WLPLQJ VWXGLHV EXW PRVWO\ LQ WKH GLUHFWLRQ RI WKH NLQG RI

IHHGEDFNWKHPRVWVLPSOHGLIIHUHQWLDWLRQFRUUHFWLYHYVQRQFRUUHFWLYHVHHIRUH[DPSOHLQ.XOKDY\DQG6WRFN>@DQGWLPLQJRIWKHIHHGEDFNDVDIXQFWLRQRIHUURUVFULWLFDOLW\VHHIRUH[DPSOHLQ)DUTXKDU>@7DNLQJLQWRDFFRXQWWKLVIHHGEDFNVGHOLYHU\RSWLRQVLQFUHDVHVRPHUHVHDUFKHUV'HPSVH\HWDO>@FDOOIRUUHYLHZLQJWKHWKHRU\H[SODLQLQJZKHQGHOD\HGIHHGEDFNFDQKHOSOHDUQHUV7KHH[SHULPHQWUHSRUWHGEHORZLVVXSSRVHGWRVKHGPRUHOLJKWRQWKHGLVFXVVHGLVVXHV

7/29/2019 7-delayedfeedback

6/19

(;3(5,0(177KHRUHWLFDOEDFNJURXQG7KHPRGHOXVHGE\+RJDUWKHWDO>@KDVEHHQDGDSWHGWRWKH QHHG RI WKLV VWXG\ 7KHUHIRUH WKHRUHWLFDOO\ WKH H[SHULPHQW FRQVLVWHG RI WZRPDLQPDQLSXODWLRQV )LUVW RI DOO DV LQ +RJDUWK HW DO >@ WKH H[DFWLQJQHVV RI OHDUQLQJHQYLURQPHQWKDVEHHQYDULHG([DFWLQJQHVVFDQKDYHERWKSRVLWLYHDQGQHJDWLYHLQIOXHQFHRQOHDUQLQJSHUIRUPDQFHDQGVWUDWHJ\VHDUFK,IWKHOHDUQLQJHQYLURQPHQWLVYHU\OHQLHQWGHFLVLRQPDNHUUHFHLYLQJSRVLWLYHIHHGEDFNUHJDUGOHVVRIWKHDFWLRQKHKDVMXVWWDNHQDVDFRQVHTXHQFH RI QRW EHLQJ SHQDOL]HG IRU HUURUV FRPPLWWHG OHDUQLQJ LV SUDFWLFDOO\LPSRVVLEOHDQGWKHVHDUFKIRUDOWHUQDWLYHVWUDWHJLHVPLQLPL]HG2QWKHRWKHUKDQGZKHQWKHOHDUQLQJHQYLURQPHQWLVH[DFWLQJGHFLVLRQPDNHUUHFHLYLQJQHJDWLYHIHHGEDFNPRVWRIWKHWLPHVVWHPPLQJIURPHDFKHUURUEHLQJVHYHUHO\SHQDOL]HGOHDUQLQJLVERRVWHGWKURXJKDOWHUQDWLYHVWUDWHJ\VHDUFKDWWKHVDPHWLPHDIIHFWLQJWKHSHUIRUPDQFHQHJDWLYHO\DWOHDVWLQVKRUWWHUP

6HFRQGRIDOOWKHYDULDEOHXVHGE\+RJDUWKHWDO>@LQFHQWLYHV KDVEHHQUHSODFHGZLWKIHHGEDFNVWLPLQJ([DFWO\KDOIRIWKHVXEMHFWVUHFHLYHGWKHLUIHHGEDFNLPPHGLDWHO\DIWHUPDNLQJHDFKGHFLVLRQWKHRWKHUKDOIVDZWKHLUIHHGEDFNZLWKDGHOD\

7KHWDVNXVHGLQWKHH[SHULPHQWUHVHPEOHGPDQ\VLWXDWLRQVIURPHYHU\GD\OLIHZKHQVXEMHFWVPXVWOHDUQIURPRXWFRPHIHHGEDFN2YHUDVHULHVRISHULRGVVXEMHFWVZHUHJLYHQD VLJQDOLQJYDULDEOH:RQ WKHEDVLVRIZKLFKWKH\ZHUH VXSSRVHGWRSUHGLFWDFULWHULRQ ZKLFK ZDV SUREDELOLVWLFDOO\ FRQQHFWHG WR WKH VLJQDO $IWHU WKHLU GHFLVLRQVXEMHFWV KDYH EHHQ LQIRUPHG DERXW WKHLU SHUIRUPDQFH KRZHYHU QRW YLD REVHUYLQJ WKHFRUUHFW YDOXH RI WKH GHFLVLRQ YDULDEOH EXW WKURXJK REVHUYLQJ DQRWKHU YDULDEOH 3HUIRUPDQFH(YDOXDWLRQ0HDVXUH 3(07KH VFDOHDQGWKHXSSHUDQGORZHU ERXQGRI3(0ZHUHQRWNQRZQWRWKHVXEMHFWVWKHUHZHUHRQO\LQIRUPHGWKDWWKHELJJHULWLVWKH

EHWWHUWKH\KDYHSHUIRUPHG7KHH[SHULPHQWKDVEHHQEDVHGRQWKHIROORZLQJSUHGLFWLRQV

+\SRWKHVLV ZKHQ OHDUQLQJ HQYLURQPHQW LV OHQLHQW HUURUV DUH QRWKHDYLO\SXQLVKHGDVDFRQVHTXHQFHRIZKLFKJHQHUDOO\SRVLWLYHIHHGEDFNLVREVHUYHGSRVWSRQHGIHHGEDFNDLGVOHDUQLQJSURFHVVWRWKHGHWULPHQWRIWKHSHUIRUPDQFHDWOHDVWLQWKHVKRUWUXQ

7KHILUVWK\SRWKHVLVLVGHULYHGIURPWKHIDFWWKDWJHQHUDOO\SRVLWLYHIHHGEDFNUHLQIRUFHVPDLQWDLQLQJRIH[LVWLQJVWUDWHJ\,IVRWKHUHLVDULVNWKDWQROHDUQLQJSURFHVVRFFXUVRQHPLJKWJHWVWXFNLQDORFDORSWLPXP3RVWSRQLQJRIWKHIHHGEDFNLQWKHVHFLUFXPVWDQFHVFRXOGIRUFHWKHVXEMHFWWRUHQGHUKLJKHULQWHOOHFWXDOHIIRUWXVHWKHDYDLODEOHLQIRUPDWLRQ

WRDKLJKHUH[WHQWDQGJUDGXDOO\DVWKHWDVNGHYHORSVLQFUHDVHWKHDELOLW\RIWKLQNLQJDERXWKHUDQVZHUVDQGVHOIFRUUHFWPLVFRQFHSWLRQVSULRUWRUHFHLYLQJIHHGEDFN

+\SRWKHVLV:KHQOHDUQLQJHQYLURQPHQWLVH[DFWLQJSHQDOWLHVLPSRVHGIRU HUURUV DUH VHYHUH WKHUHIRUH JHQHUDOO\ QHJDWLYH IHHGEDFN LV REVHUYHGSRVWSRQHGIHHGEDFNDGYHUVHO\DIIHFWVOHDUQLQJSURFHVVDQGSHUIRUPDQFH

7KHVHFRQGK\SRWKHVLVLVEDVHGRQWKHLGHDWKDWERWKYHU\H[DFWLQJHQYLURQPHQWDQGGHOD\HGIHHGEDFNZRXOG WXUQWKH OHDUQLQJHQYLURQPHQW LQWRDYHU\ZLFNHGRQH6LQFHJLYHQJHQHUDOO\QHJDWLYHIHHGEDFNVXEMHFWVWHQGWRVHHNIRUDOWHUQDWLYHVWUDWHJLHVDOUHDG\

7/29/2019 7-delayedfeedback

7/19

WKHGHOD\LQIHHGEDFNLQIRUPDWLRQFDQRQO\FRQIRXQGWKHLQIHUHQWLDODQGHYDOXDWLYHUROHRIIHHGEDFNWKHUHIRUHGHFUHPHQWLQJWKHOHDUQLQJSURFHVVDQGSHUIRUPDQFH

$GGLWLRQDOO\WKHIROORZLQJZRUNLQJK\SRWKHVHVKDYHEHHQGUDZQ$VLQ+RJDUWKHWDO>@LWZDVFRQMHFWXUHGWKDWWKHSHUIRUPDQFHLVDVLQJOHSHDNHGLQYHUWHG8VKDSHGIXQFWLRQRIH[DFWLQJQHVV%DVLQJRQ.XOKDY\DQG$QGHUVRQ>@ILQGLQJVWKDWVXEMHFWVJLYHQGHOD\HGIHHGEDFNVSHQGPRUHWLPHVWXG\LQJWKHIHHGEDFNWKHVDPHZDVVXSSRVHG 7KH VXEMHFWV ZHUH H[SHFWHG WR GHFODUH ZRUVH PRRG ZLWK LQFUHDVLQJ OHYHOV RIH[DFWLQJQHVV$WWKH VDPH WLPHVXEMHFWVZHUH VXSSRVHG WRGHFODUH RQDYHUDJHZRUVHPRRGIRUGHOD\HGIHHGEDFNWKDQIRULPPHGLDWHIHHGEDFNFRQGLWLRQ'HWDLOHGGHVLJQ6XEMHFWV7KHVXEMHFWVZHUHDOOJUDGXDWHVWXGHQWVIURPWKH0DVWHUDQG'RFWRUDOSURJUDP LQ(FRQRPLFV )LQDQFHDQG%XVLQHVV DW WKH8QLYHULVWDW3RPSHX

)DEUD $OO WRJHWKHU VXEMHFWV ZHUH UHFUXLWHG YLD DQ HPDLO PHVVDJH LQIRUPLQJ DOOVWXGHQWVRIWKHGHSDUWPHQWRI(FRQRPLFVDQG%XVLQHVVDERXWDSRVVLELOLW\RISDUWLFLSDWLRQLQ D UHPXQHUDWHGH[SHULPHQWRQ GHFLVLRQPDNLQJ

0HDQ DJH RI WKH VXEMHFWVZDV

\HDUV7KHUHZHUHPDOHVDQGIHPDOHVLQWKHVDPSOH6XEMHFWVZHUHUHPXQHUDWHGZLWKHXURVIRUWKHLUSDUWLFLSDWLRQEXWQRWIRUSHUIRUPDQFH3URFHGXUH$WWKHEHJLQQLQJVXEMHFWVZHUHLQVWUXFWHGWKDWWKHH[SHULPHQWFRQVLVWHGRIURXQGV DQG WKDW LQ HDFK URXQG WKH\ KDG WR PDNH GHFLVLRQV 7KH\ ZHUH DVNHG WRFRPSOHWHD TXHVWLRQQDLUH DIWHUHDFKURXQG7KH\ZHUHLQIRUPHG WKDW WKHUHZDVQRWLPHFRQVWUDLQW WR ILQLVK WKH WDVN EXW WKDW WLPH ZDV JRLQJ WR EH PHDVXUHG 6XEMHFWV ZHUHDOORZHG WR WDNH QRWHV HYHQ WKRXJK WKH VRIWZDUH NHSW WUDFN RI WKHLU GHFLVLRQV LQ HDFK

URXQGDQGWKHVXEMHFWVFRXOGEURZVHWKHLUSUHYLRXVDQVZHUVZLWKRXWDQ\UHVWULFWLRQV$OOVXEMHFWVZHUHDVNHGWRUHDGFDUHIXOO\WKHIROORZLQJLQVWUXFWLRQ

,WLVVXPPHU

7/29/2019 7-delayedfeedback

8/19

WKDWRWKHUZLVHZRXOGQHYHUEHDYDLODEOHWR\RXLHWKHDFWXDOGHPDQGIRUHDFKGD\@

6XEMHFWVZHUHUDQGRPO\DVVLJQHG

WRRQHRIWKHVL[JURXSVFUHDWHGE\FURVVLQJWZRYDULDEOHVRQHZLWKWKUHHOHYHOVH[DFWLQJYVLQWHUPHGLDWHYVOHQLHQWHQYLURQPHQWDQGWKHRWKHURQHZLWKWZROHYHOVLPPHGLDWHYVGHOD\HGIHHGEDFN

$W WKH EHJLQQLQJRIHDFK SHULRGD VLJQDO: ZHDWKHU IRUHFDVW WHPSHUDWXUH LQ&HOVLXVGHJUHHVZDVUDQGRPO\FKRVHQHDFKWLPHIURPDQRUPDOGLVWULEXWLRQZLWKPHDQDQGVWDQGDUGGHYLDWLRQ7KHWHPSHUDWXUHZDVWKDQWUDQVODWHGLQWR)DKUHQKHLWGHJUHHV

DQGSUHVHQWHGWRWKHVXEMHFW,PPHGLDWHO\DIWHUREVHUYLQJWKHVLJQDOVXEMHFWVKDGWRWDNHD SURGXFWLRQ GHFLVLRQ YDOXHV UHVWULFWHG WR WKH QDWXUDO LQWHUYDO ! 6XEMHFWVIROORZLQJ WKH LPPHGLDWH IHHGEDFN FRQGLWLRQ UHFHLYHG WKHLU 3(0 DQG DX[LOLDU\YDULDEOHV LPPHGLDWHO\ 6XEMHFWV IROORZLQJ WKH GHOD\HG IHHGEDFN FRQGLWLRQ UHFHLYHGWKHLUSHUIRUPDQFHHYDOXDWLRQPHDVXUHZLWKDGHOD\RIWZRSHULRGVQ

%DVHG RQ (YDOXDWLRQ SRLQWV IURP +RJDUWK HW DO >@ WKH 3HUIRUPDQFH (YDOXDWLRQ0HDVXUH3(0ZDVFDOFXODWHGXVLQJWKHIRUPXOD

'XHWRDVSHFLILFLQGLYLGXDODSSOLFDWLRQRIWKHVRIWZDUHVXEMHFWVZHUHDOVRLQIRUPHGWKDWGXULQJWKHWLPHWKH\ILOOLQWKHILUVWTXHVWLRQQDLUHWKHH[SHULPHQWHUZRXOGPDQXDOO\UHVWDUWWKHJDPHIRUWKHP7KLVPDQLSXODWLRQKDVEHHQGRQHLQRUGHUWRPDNHWKHWDVNVRPHZKDWPRUHGLIIFXOW7KHFRQMHFWXUHZDVWKDWPRVWRIWKHVXEMHFWVZHUHQRWJRLQJWRNQRZWKH)DKUHQKHLWVFDOHVRLPPHGLDWHRSWLPDOVFDOLQJRIWKHWDVNZRXOGEHLPSRVVLEOH,QSUDFWLFHRQO\RXWRIVXEMHFWVNQHZWKHUHODWLRQVKLSEHWZHHQWKH)DKUHQKHLWDQG&HOVLXVVFDOH0RUHVWULFWVWDWLVWLFDODQDO\VLVZDVQRWSRVVLEOHEXWWKHDXWKRUKDGQRWIRXQGVLJQVRIVXEMHFWVNQRZLQJWKHGLIIHUHQFHEHWZHHQWKHWZRVFDOHVFRQVLVWHQWO\RXWSHUIRUPLQJWKHVXEMHFWVZKRGLGQRWNQRZLW,QWKHILUVWWZRSHULRGVVXEMHFWVZHUHLQIRUPHG1RSHUIRUPDQFHHYDOXDWLRQDYDLODEOH\HW6WDUWLQJIURPSHULRGWKHIROORZLQJPHVVDJHZDVGLVSOD\HG

7/29/2019 7-delayedfeedback

9/19

3(04'

:KHUH LV WKH H[DFWLQJQHVV SDUDPHWHU UHVSHFWLYHO\ IRU OHQLHQW

LQWHUPHGLDWHDQGH[DFWLQJHQYLURQPHQW4LVVXEMHFWVUHVSRQVHDQG'LVFRUUHFWYDOXHRIWKHFULWHULRQ$VFDQEHVHHQ3(0IXQFWLRQLVDQHJDWLYHIXQFWLRQRIVTXDUHGGLVFUHSDQF\EHWZHHQWKH

VXEMHFWV DQVZHUDQGWKHFRUUHFW DQVZHU7KHSDUDPHWHUZDV XVHG WRPDQLSXODWH WKH

H[DFWLQJQHVV RI WKH OHDUQLQJ HQYLURQPHQW LQFUHDVLQJ LQFUHDVHV WKH SHQDOW\ IRU

HUURQHRXVUHVSRQVH

'KDGDVWURQJFRUUHODWLRQZLWKWKHZHDWKHUVLJQDO:DQGZDVGHVFULEHGDVDOLQHDU

IXQFWLRQ':HUURUWHUP

:KHUHHUURUWHUPLVDVPDOOQRLVHZLWKPHDQRI]HUR

7RFRPSDUHSHUIRUPDQFH3 DFURVVGLIIHUHQW OHYHOVRIH[DFWLQJQHVV WKH REVHUYHGSHUIRUPDQFHLQDJLYHQWULDOLZDVPHDVXUHGE\

3L_4L'L_

ZLWKPHDQSHUIRUPDQFHRYHUQWULDOVFDOFXODWHGXVLQJWKHIROORZLQJIRUPXOD

Q3L

6LQFH LV DPHDQ SHUIRUPDQFH RI MXVW RQH VXEMHFW PHDQ SHUIRUPDQFH LQ D

JURXSZDVFDOFXODWHGRYHUPVXEMHFWVLQHDFKJURXS

P

$GGLWLRQDOO\DX[LOLDU\YDULDEOHV$DQG%ZHUHFDOFXODWHGLQWKHIROORZLQJZD\

,I4'WKHQ$4DQG%,I4!'WKHQ$'DQG%4'

9DULDEOHV$DQG%KDYHEHHQLQWURGXFHGLQRUGHUWRSURYLGHXVHIXOSLHFHRILQIRUPDWLRQIRUVXEMHFWVZLWKKLJKXQGHUVWDQGLQJRIWKHV\VWHP3RVWH[SHULPHQWDOSURFHGXUH$IWHUHDFKURXQGRIWKHH[SHULPHQWVXEMHFWVZHUHDVNHGWRFRPSOHWHD VKRUW TXHVWLRQQDLUH7KLVTXHVWLRQQDLUHZDV GHVLJQHG LQRUGHU WRFKHFN WKHGHVLJQHUNQRZOHGJHWKDWVXEMHFWVDFTXLUHGGXULQJWKHH[SHULPHQW

7/29/2019 7-delayedfeedback

10/19

4XHVWLRQVDIWHUILUVWURXQG

$JH6H[3OHDVH LQGLFDWHRQ WKHVFDOHEHORZ/LNHUWVFDOH IURPZKHUH LVWRWDOO\UHOD[HGDQGLVYHU\VWUHVVHGKRZ\RXIHOWGXULQJWKLVURXQGRIWKHH[SHULPHQW"DIWHUVHFRQGURXQG3OHDVHLQGLFDWHRQWKHVFDOHEHORZ/LNHUWVFDOHIURPZKHUHLVWRWDOO\UHOD[HGDQGLVYHU\VWUHVVHGKRZ\RXIHOWGXULQJWKLVURXQGRIWKHH[SHULPHQW"7U\WRH[SODLQLQJHQHUDOWHUPV KRZGRHVWKHJDPHZRUN")RFXVVSHFLDOO\RQWKHUROHVSOD\HGE\GLIIHUHQWYDULDEOHV 7HPSHUDWXUH $FWXDO GHPDQG DQG 3URGXFWLRQOHYHODQGWKHLULQWHUUHODWLRQV:KDWKDYH\RXOHDUQHGDERXWDX[LOLDU\YDULDEOHV$DQG%"&DQ\RXWKLQNRIDQ\VSHFLILFWLSV\RXZRXOGJLYHWRDQRWKHUSHUVRQLQRUGHUWRKHOSKHUWRDFKLHYHKLJKSHUIRUPDQFHHYDOXDWLRQPHDVXUH"

%HIRUH WKH H[SHULPHQW GLG\RX NQRZ WKH UHODWLRQEHWZHHQ&HOVLXV DQG )DKUHQKHLWGHJUHHV"

7/29/2019 7-delayedfeedback

11/19

7DEOH3HUIRUPDQFHDQGLQFRQVLVWHQF\LQOHQLHQWPHGLXPDQGH[DFWLQJHQYLURQPHQW

,PPHGLDWHIHHGEDFN 'HOD\HGIHHGEDFN

5RXQG /HQLHQW 0HGLXP ([DFWLQJ /HQLHQW 0HGLXP ([DFWLQJ3HUIRUPDQFH

0 6' 0 6' $YHUDJH

,QFRQVLVWHQF\

5VTXDUH

$YHUDJH

)LJXUH0HDQSHUIRUPDQFHE\W\SHVRIHQYLURQPHQWVDQGURXQG

5RXQGV

0HDQSHUIRUPDQFH

/HQLHQW

0HGLXP

([DFWLQJ

,Q WHUPVRIPRUH VWUXFWXUHG VWDWLVWLFDO DQDO\VLV LQLWLDO$129$UHSRUWHG WKDW WKHRQO\ VLJQLILFDQW HIIHFW LV WKH LQWHUDFWLRQ EHWZHHQ YDULDEOHV )HHGEDFN DQG /HDUQLQJHQYLURQPHQW +RZHYHU WKH WHVWV IRU KRPRJHQHLW\ RI YDULDQFH IRU PHDQV VKRZHGVWDWLVWLFDOO\ VLJQLILFDQW GLIIHUHQFHV EHWZHHQ WKHYDULDQFHV ERWK%DUWOHWWV DQG /HYHQHVWHVWV VLJQLILFDQW DW 7KHUHIRUH WKLV UHVXOW PLJKW EH TXHVWLRQDEOH 6LQFH WKH

KRPRJHQHLW\RIYDULDQFHWHVWVKRZHGWKDWYDULDQFHVGLIIHUSUHGRPLQDQWO\LQWKHOHQLHQW

7/29/2019 7-delayedfeedback

12/19

HQYLURQPHQWEHWZHHQLPPHGLDWHDQGGHOD\HGIHHGEDFNFRQGLWLRQVDQGQRWPXFKLQRWKHUHQYLURQPHQWV,SHUIRUP$129$ZLWKURXQGVDVDUHSHDWHGIDFWRURQO\IRUPHGLXPDQGH[DFWLQJOHDUQLQJHQYLURQPHQW)RUPHGLXPHQYLURQPHQW$129$VKRZVDVLJQLILFDQWGLIIHUHQFHEHWZHHQPHDQVRIURXQGV)06HS)RUH[DFWLQJHQYLURQPHQWWKHHIIHFWIRUURXQGVLVVLJQLILFDQW)06HS)RUGLIIHUHQFHVEHWZHHQWKHWLPLQJRIIHHGEDFNWKHQXOOK\SRWKHVLVRIPHDQVEHLQJIURPWKHVDPHGLVWULEXWLRQLVMXVWDWWKHUHMHFWLRQUHJLRQ)06HS

)LJXUH0HDQSHUIRUPDQFHIRUURXQGVDQGE\H[SHULPHQWDOFRQGLWLRQV

OHQLHQWPHGLXP

H[DFWLQJ

LPPHGLDWH

GHOD\HG

0HDQ

SHUIRUPDQFH

/HDUQLQJHQYLURQPHQW

)HHGEDFN

6LQFH VWDQGDUG GHYLDWLRQV UHSRUWHG LQ 7DEOH DUH MXVW LQGLFDWRUV RI JURXSVKRPRJHQHLW\ZHQHHGDQRWKHUPHDVXUHRIVXEMHFWVTXHVWIRUVWUDWHJ\/HWXVLPDJLQHDPRGHOZKHUHDOODQVZHUVJLYHQE\HDFKVXEMHFWVDUH UHJUHVVHGRQ WKH VLJQDOWKH\KDYHREVHUYHGWKDWLV

4:]5VTXDUHGREWDLQHGWKURXJKWKLVUHJUHVVLRQFDQWHOOXVKRZPXFKRIWKHYDULDELOLW\RIWKHUHVSRQVHVFDQEHH[SODLQHGE\WKHPRGHOLHKRZZHOOWKHWDVNLVXQGHUVWRRGDQGKRZ

FRQVLVWHQWWKHVXEMHFWVDUHZLWKH[HFXWLQJWKHLUFXUUHQWVWUDWHJLHV7KHUHIRUH5VTXDUHGJLYHVXVDPHDVXUHRILQFRQVLVWHQF\LHKRZPXFKVXEMHFWVGHYLDWHIURPHDUOLHUFKRVHQVWUDWHJ\

,UHSRUWWKLVPHDVXUHDYHUDJHGRYHUDOOVXEMHFWVLQHDFKH[SHULPHQWDOJURXS

DV,QFRQVLVWHQF\LQ7DEOH8QGHULPPHGLDWHIHHGEDFNDVSUHGLFWHGZHFDQVHHWKDWLQFRQVLVWHQF\LQFUHDVHV

ZLWK LQFUHDVLQJ H[DFWLQJQHVV 7KLV LV KRZHYHU QRW WKH FDVH IRU GHOD\HG IHHGEDFN7KHGLIIHUHQFH LQ WHUPV RI LQFRQVLVWHQF\ EHWZHHQ GHOD\HG DQG LPPHGLDWH IHHGEDFN LQ WKHPHGLXPH[DFWLQJHQYLURQPHQWLVQRWELJH[FHSWRIWKHILUVWURXQGEXWWKLVLVLQWXLWLYHO\

5VTXDUHGIURPUHJUHVVLRQPRGHOVKDVEHHQZLGHO\XVHGLQWKHMXGJPHQWUHVHDUFKDQGOHQVPRGHOVHHIRUH[DPSOH6WHZDUW>@

7/29/2019 7-delayedfeedback

13/19

HDV\WRH[SODLQ6XEMHFWVLQWKHGHOD\HGIHHGEDFNJURXSKDGUHODWLYHO\OHVVLQIRUPDWLRQLQWKHILUVWURXQGVRWKH\ IRXQGRXWWKHLUVWUDWHJ\ ODWHUWKDQVXEMHFWVLQWKHLPPHGLDWHIHHGEDFN JURXS %\ WKH VHFRQG URXQG GHOD\HG VXEMHFWV DOUHDG\ RXWSHUIRUPHG WKHLPPHGLDWHJURXS,QWHUPVRIPRUH VWULFW VWDWLVWLFDODQDO\VLV WZRZD\$129$ZLWKURXQGV DVUHSHDWHGIDFWRU VKRZHG WKDW WKHRQO\VLJQLILFDQW HIIHFWLV WKHRQHIRU URXQGV)0VHS

7KH LQWHUHVWLQJ FDVH LV WKH RQH RI OHQLHQW DQG H[DFWLQJ HQYLURQPHQW ,Q WKHH[DFWLQJHQYLURQPHQWVXEMHFWVUHFHLYLQJGHOD\HGIHHGEDFNIURPWKHEHJLQQLQJVKRZHGOHVVLQFRQVLVWHQF\DQGRXWSHUIRUPHGVXEMHFWVUHFHLYLQJLPPHGLDWHIHHGEDFNVLJQLILFDQWO\$129$ UHYHDOHG WKDW ERWK HIIHFWV DUH VWDWLVWLFDOO\ VLJQLILFDQWIRU URXQGV ) 0VHSDQGIRUFROXPQV)0VHS,QWKHOHQLHQW HQYLURQPHQW VXEMHFWV UHFHLYLQJ GHOD\HG IHHGEDFN IURP WKH EHJLQQLQJ VWDUWHGSHUIRUPLQJ EDGO\ DQG E\ WKH VHFRQG URXQG QRW RQO\ GLG QRW LPSURYH EXW DOVR ZHUHLQFUHDVLQJO\ LQFRQVLVWHQW LQ WKHLU UHVSRQVHV $129$ VKRZHG VLJQLILFDQFH RI WKH

IHHGEDFNVWLPLQJFKDQJH) 0VHSEXWQRWIRUUHSHDWHGURXQGV $GGLWLRQDO UHVXOWV 7LPH 7KH WLPH RI HDFK GHFLVLRQ KDV EHHQ PHDVXUHG 7KHDYHUDJHGWLPHIRUH[SHULPHQWDOJURXSVSHUURXQGDUHUHSRUWHGLQ7DEOH,WFDQEHVHHQWKDWRQDYHUDJHVXEMHFWV UHFHLYLQJ GHOD\HG IHHGEDFN WRRNPRUH WLPH LQRUGHUWRPDNHGHFLVLRQV$129$ZLWKURXQGVDVDUHSHDWHGIDFWRUUHYHDOHGVLJQLILFDQFHIRUH[DFWLQJHQYLURQPHQW)06HSDQGIRUPHGLXPOHDUQLQJHQYLURQPHQW)06HSEXWQRWIRUOHQLHQWHQYLURQPHQW

7DEOH7LPHRIGHFLVLRQPDNLQJE\URXQGLQDOOH[SHULPHQWDOFRQGLWLRQV

,PPHGLDWHIHHGEDFN 'HOD\HGIHHGEDFN

5RXQG /HQLHQW 0HGLXP ([DFWLQJ /HQLHQW 0HGLXP ([DFWLQJ 7LPH 6' 7LPH 6' $YHUDJH

6WUHVV2QHRIWKHTXHVWLRQVDVNHGLQWKHTXHVWLRQQDLUHVDIWHUHDFKURXQGZDV 3OHDVHLQGLFDWHRQWKHVFDOHEHORZ/LNHUWVFDOHIURPZKHUHLVWRWDOO\UHOD[HGDQGLVYHU\VWUHVVHG KRZ \RX IHOW GXULQJ WKLV URXQG RI WKH H[SHULPHQW"7DEOH UHSRUWV PHDQDQVZHUVSHUJURXSDQGWKHHYROXWLRQRYHUWKHURXQGV

7/29/2019 7-delayedfeedback

14/19

7DEOH$XWRHYDOXDWLRQRIVWUHVVRYHUURXQGVLQDOOH[SHULPHQWDOFRQGLWLRQV

,PPHGLDWHIHHGEDFN 'HOD\HGIHHGEDFN

5RXQG /HQLHQW 0HGLXP ([DFWLQJ /HQLHQW 0HGLXP ([DFWLQJ

$VFDQEHVHHQWKHUHDUHQRELJGLIIHUHQFHVEHWZHHQWKHJURXSVUHFHLYLQJLPPHGLDWHDQGGHOD\HGIHHGEDFNH[FHSWRIPHGLXPH[DFWLQJOHDUQLQJHQYLURQPHQW'HVLJQHUV NQRZOHGJH 7KHUHVWRI WKH TXHVWLRQV LQ WKH TXHVWLRQQDLUHZDVVXSSRVHG WRJDXJHVXEMHFWV GHVLJQHU NQRZOHGJHRI WKH WDVN DIWHUKDYLQJ WDNHQGHFLVLRQV7KHTXHVWLRQV KDG EHHQ HYDOXDWHG DQG D SRLQW KDG EHHQ JLYHQ IRU HDFK YDOXDEOH SLHFH RILQIRUPDWLRQ UHSRUWHGE\D VXEMHFW)RU TXHVWLRQ JHQHUDO NQRZOHGJH SRLQWVPLJKWKDYHEHHQ DFKLHYHG E\ VWDWLQJ WKH VLJQ DQG WKH VORSH RI WKH UHODWLRQ EHWZHHQ: DQGGHFLVLRQYDULDEOHEHLQJDZDUHRIWZRW\SHVRIHUURUVRYHUHVWLPDWLQJRUXQGHUHVWLPDWLQJWKH SURGXFWLRQ OHYHO H[SUHVVLQJ WKH XSSHU OLPLWRI WKH SHUIRUPDQFHPHDVXUHHWF )RUTXHVWLRQNQRZOHGJHDERXWYDULDEOHV$DQG%SRLQWVZHUHJLYHQIRUVWDWLQJWKDW$% GHFLVLRQ YDULDEOH LQGLFDWLQJ WKDW YDULDEOHV$DQG%GHSHQGRQ WKH WZR W\SHVRIHUURUHWF)LQDOO\IRUTXHVWLRQVSHFLILFWLSVSRLQWVZHUHJLYHQIRUDQ\YDOLGDGYLFHOLNHIRUH[DPSOHWKDWLWLVDOZD\VEHWWHUWRRYHUHVWLPDWHWKHJXHVVDQGVRRQ7KHPHDQVFRUHVREWDLQHGE\WKHVXEMHFWVLQHDFKH[SHULPHQWDOJURXSDUHUHSRUWHGLQ7DEOH

7DEOH'HVLJQHUVNQRZOHGJH VFRUHVLQDOOH[SHULPHQWDOFRQGLWLRQV

/HDUQLQJHQYLURQPHQW

/HQLHQW 0HGLXP ([DFWLQJ ,PPHGLDWHIHHGEDFN 'HOD\HGIHHGEDFN

+HUHDJDLQGDWDVKRZVVLPLODUOHYHORIXQGHUVWDQGLQJRIWKHWDVNZLWKDQH[FHSWLRQRIDFOHDULQGLFDWLRQRIWKHGLIIHUHQFHEHWZHHQSRRUO\SHUIRUPLQJGHOD\HGIHHGEDFNOHQLHQWHQYLURQPHQW JURXS DQG EHVW SHUIRUPLQJ GHOD\HG IHHGEDFN PHGLXP HQYLURQPHQWJURXS

',6&866,21

7KH WZR PDLQ K\SRWKHVHV KDYH QRW EHHQ FRQILUPHG E\ WKH H[SHULPHQWDO GDWD'HOD\LQJIHHGEDFNLQWKHOHQLHQWHQYLURQPHQWGLGQRWKHOSWKHGHFLVLRQPDNHUVZKHUHDV

7/29/2019 7-delayedfeedback

15/19

GHOD\LQJ IHHGEDFN XQGHU H[DFWLQJ HQYLURQPHQW GLG ,Q WHUPV RI DGGLWLRQDO ZRUNLQJK\SRWKHVHV SHUIRUPDQFH ZDV D VLQJOHSHDNHG LQYHUWHG8VKDSHG IXQFWLRQ RIH[DFWLQJQHVV6XEMHFWVJLYHQGHOD\HGIHHGEDFNVSHQWPRUHWLPHRQWKHGHFLVLRQPDNLQJEXW WKH\ GLG QRW UHSRUW VLJQLILFDQWO\ PRUH VWUHVV WKDQ VXEMHFWV REVHUYLQJ LPPHGLDWHIHHGEDFN 7KH PRRG RI VXEMHFWV ZDV DOVR UREXVW WR LQFUHDVHV RI WKH H[DFWLQJQHVV RIOHDUQLQJHQYLURQPHQWV

7KHUHDUHWKUHHGLIIHUHQWZD\VRIDUJXLQJLIOHDUQLQJSURFHVVRFFXUUHG7KHILUVWRQH LV MXVW YLD REVHUYLQJ WKH SHUIRUPDQFH LQGLFDWHG LQ)LJXUH E\FOHDUO\ LQFUHDVLQJFXUYHV 8QGHU WKLV DSSURDFK GHOD\LQJ IHHGEDFN LQ OHQLHQW HQYLURQPHQW LQVWHDG RIERRVWLQJ OHDUQLQJ SURFHVV LPSDFWHG LW QHJDWLYHO\ 7KH VHFRQG ZD\ LV WR WDNH LQWRFRQVLGHUDWLRQWKHGDWDFROOHFWHGLQTXHVWLRQQDLUHV7KHHYDOXDWLRQRIWKLVGDWDLVKRZHYHUVXEMHFWLYH DQGLW GRHVQRW KHOS WRUHDFKDQ\ FRQVLVWHQW VWDQGSRLQW7KH WKLUGZD\ LVWRVHDUFK IRU VLJQV RI GHVLJQHUV NQRZOHGJH LQ VXEMHFWV UHVSRQVHV 2QFH D VXEMHFWXQGHUVWRRG WKDW ZKHQHYHU KLV SURGXFWLRQ LV RYHUHVWLPDWHG YDULDEOH $ JLYHV KLP WKH

FRUUHFWDQVZHUIRUFXUUHQWWHPSHUDWXUHDQGJLYHQWKDWWKHWHPSHUDWXUHVWHQGWRUHSHDWVKHVKRXOGXVHWKHFRUUHFWDQVZHUHYHU\WLPHSUHVHQWHGZLWKWKHVDPHVLJQDO$OODQVZHUVRIHDFKRQHRI VXEMHFWVKDGEHHQDQDO\]HGVHDUFKLQJIRU WKLVSDWWHUQ8QIRUWXQDWHO\WKLV PHWKRG SUHVHQWHG QR GLVFULPLQDWLYH SRZHU VLQFH DQ DYHUDJH RI WZR VXEMHFWV SHUJURXSIROORZLQJWKLVUXOHKDYHEHHQIRXQG

,QWHUPVRIVWUDWHJ\VHDUFKLQWHUHVWLQJSKHQRPHQDRFFXUUHG)LUVWRIDOOVXEMHFWVLQWKHOHQLHQWOHDUQLQJHQYLURQPHQWZKLOHSUHVHQWHGZLWKGHOD\HGIHHGEDFNWHQGHGWRUHUHSUHVHQWWKHWDVNPRUHRIWHQWKDQWKHVXEMHFWVUHFHLYLQJLPPHGLDWHIHHGEDFNWKHUHIRUHFKDQJLQJWKHVWUDWHJ\WKH\ZHUHDSSO\LQJPRUHRIWHQ2QUHYHUVHGHOD\LQJWKHIHHGEDFNLQWKHH[DFWLQJHQYLURQPHQWPDGHWKHVXEMHFWVWREHPRUHFRQVLVWHQWLQWKHDSSOLFDWLRQRIWKHLU VWUDWHJ\ , RIIHU WZR ZD\V RI H[SODLQLQJ WKHVH ILQGLQJV 2QH SRVVLEOH ZD\ RI

UHDVRQLQJLVWKDWLQWKHOHQLHQWHQYLURQPHQWVXEMHFWVVHDUFKHGIRULPSURYLQJWKHTXDOLW\RI GHOD\HG IHHGEDFN WKURXJK H[FHVV YDULDELOLW\ 6XEMHFWV DIWHU KDYLQJ WDNHQ ILUVWGHFLVLRQVZLWKRXWDQ\NQRZOHGJHDERXWWKHWDVNKDGDORWRIFKDQFHVWRUHFHLYHDSRVLWLYHRUVOLJKWO\QHJDWLYH IHHGEDFN )URP WKHPRPHQW D UHSUHVHQWDWLRQ RI WKH WDVN KDVEHHQPDGHWKHGHOD\RI3(0UHTXLUHG WKHVXEMHFWVWR WHVWWKHLUVWUDWHJLHV LQDPRUHH[WUHPHZD\ LQ D VHQVH RYHUUHDFWLQJ WR WKH VLJQDO LQRUGHU WR FKHFN WKHLU YDOLGLW\ 6LQFH WKHIHHGEDFNSHUVHZDVRIDSUHWW\EDGTXDOLW\LHWRDJUHDWH[WHQWHYHU\DFWLRQUHFHLYHGDSRVLWLYHIHHGEDFNGHOD\LQJLWVYLVXDOL]DWLRQIRUFHGVXEMHFWVWRLQFUHDVHWKHYDULDELOLW\RIDQVZHUVWKHUHIRUHLPSURYLQJWKHLQIHUHQWLDOFRPSRQHQWRIWKHLQIRUPDWLRQEHLQJIHGEDFNWRWKHP,QWKHH[DFWLQJHQYLURQPHQWVLQFHHDFKVPDOOGHYLDWLRQIURPWKHFRUUHFWDQVZHUUHVXOWHG

LQD YHU\SHQDOL]HG3(0WKLVPHFKDQLVPZDVQRWQHHGHG(YHQGHOD\HGWKHIHHGEDFNKDGHQRXJKTXDOLW\WRJXLGHWKHVXEMHFWVWRZDUGVDFRUUHFWVWUDWHJ\

$QRWKHU UHODWHG H[SOLFDWLRQ PLJKW WDNH LQWR FRQVLGHUDWLRQ DVSLUDWLRQ OHYHOV6XEMHFWVZHUHPRWLYDWHGWRIRFXVRQPD[LPL]LQJWKHLUSHUIRUPDQFHEXWWKHRQHVLQWKHOHQLHQWHQYLURQPHQWWKURXJKREVHUYLQJPDLQO\SRVLWLYH3(0PLJKWKDYHFRQIRUPHGZLWKLWPRUHHDVLO\WKDQVXEMHFWVIURPWKHH[DFWLQJHQYLURQPHQW

7KHODWWHUVXEMHFWVLQWXUQ

7KLVZD\RIUHDVRQLQJLVEDVHGPDLQO\RQWKHLQWHUYLHZZLWKRQHRIWKHVXEMHFWVLQWKHOHQLHQWHQYLURQPHQWGHOD\HGIHHGEDFNJURXSZKRZDVFRS\LQJWKHVLJQDO:LQWRWKHSURGXFWLRQGHFLVLRQFRQVLVWHQWO\RYHUDFRQVLGHUDEOHQXPEHURISHULRGV:KHQDVNHGZKHWKHUVKHXQGHUVWRRGWKHWDVNWKHVXEMHFWDQVZHUHG\HVDQGWKDWWKLVKHXULVWLFVWUDWHJ\VHHPHGWRZRUNIRUVRPHWLPH

7/29/2019 7-delayedfeedback

16/19

IURPWKHEHJLQQLQJZHUHPXFKPRUHLQFOLQHGWRVHHELJQHJDWLYHQXPEHUVWKHUHIRUHRQFHWKH\REVHUYHGDSRVLWLYH3(0WKH\ZHUHPXFKPRUHPRWLYDWHGWRIRFXVRQPDLQWDLQLQJWKHSURVSHFWLYH3(0VSRVLWLYH7KLVZD\RIWKLQNLQJLVVRPHZKDWUHODWHGWRWKHWKHRU\RI+LJJLQV>@>@VWDWLQJWKDW7KHRXWFRPHRIORVVDQGIDLOXUHLVFRQJUXHQWZLWKWKH

VWUDWHJ\RIDYRLGLQJORVVLQSUHYHQWLRQIRFXVZKHUHDVWKHRXWFRPHRIJDLQDQGVXFFHVVLVFRQJUXHQWZLWKWKHVWUDWHJ\RIDSSURDFKLQJUHZDUGV,IZHDFFHSWWKHIDFWWKDWGHOD\HGIHHGEDFN LQ WKH H[DFWLQJ HQYLURQPHQW LQFUHDVHG WKH QHJDWLYH IUDPLQJ RI WKH WDVNGHFUHDVHGYDULDELOLW\LQUHVSRQVHVPLJKWEHH[SODLQHGWKURXJKWKHIXUWKHUORVVSUHYHQWLRQDSSURDFK ,I RQ WKH RWKHU KDQG ZH DFFHSW WKDW GHOD\HG IHHGEDFN LQ WKH OHQLHQWHQYLURQPHQWLQFUHDVHGWKHIHHOLQJRIVXFFHVVWKURXJKLQFUHDVLQJWKHVDOLHQF\RISRVLWLYH3(0VLQFUHDVHGYDULDELOLW\RIUHVSRQVHVFRXOGEHH[SODLQHG

6RPHFRPPHQWVDUHLQRUGHUKHUH)LUVWRIDOOWKURXJKRXWWKHVWXG\LWLVDVVXPHGWKDWDOOWKHVXEMHFWVDUHZHOOPRWLYDWHGDQGWKHLUFKRLFHVDUHLQOLQHZLWKVRPHORQJUXQGHVLUHWRSHUIRUPZHOO$V,KDYHQR UHDVRQV WRWKLQN WKHFRQWUDU\PDUJLQDOQXPEHURI

VXEMHFWVUHSRUWHGWKDWWKH\KDYHFRPPLWWHGGDWDLQWURGXFWLRQHUURUVIH[LQWURGXFLQJLQVWHDGRI6HFRQGRIDOOLW LVZRUWKVWUHVVLQJWKDWWKHNLQGRI WDVNDQDO\]HGLQWKHH[SHULPHQW LV GLIIHUHQW WR WKH VWDQGDUG WDVNV XVHG WR DQDO\]H WKH LQIOXHQFHRI GHOD\HGIHHGEDFN7KHLFHFUHDPH[SHULPHQWLVQRWDG\QDPLFGHFLVLRQPDNLQJSURFHVVLQWKHXVXDOVHQVHWKHUHLVQRFDUU\RYHUHIIHFWIURPRQHSHULRGWRWKHQH[WKRZHYHUVXEMHFWVDUH VXSSRVHG WR PDNH GHFLVLRQV ZLWKRXW NQRZLQJ WKH RXWFRPH IHHGEDFN IURP WKHSUHYLRXV SHULRGV HYHQ GHOD\HG 7KHUHIRUH VWDQGDUG WHUPLQRORJ\ RI GHOD\HG RUSRVWSRQHGIHHGEDFNVKRXOGEHXVHGZLWKFDUH,SURSRVHWRPHUJHWKRVHWHUPVLQRQHXOWUDGHOD\HGIHHGEDFNWKDWPHDQVWKDWWKHIHHGEDFNLVGHOD\HGZLWKUHVSHFWWRWKHZKROHWDVNEXWSRVWSRQHGZLWKUHVSHFWWRWKHJLYHQWULDO

3RVVLEOHFULWLFLVP7KHUHLVDQXPEHURISRVVLEOHFULWLFLVPVWRWKHUHSRUWHGVWXG\)LUVWRIDOOWKHVDPSOHVL]HRIVXEMHFWVPDNHVDOOWKHILQGLQJVVWDWLVWLFDOO\TXHVWLRQDEOH6HFRQGRIDOOWKHOLQHDUUHODWLRQVKLSEHWZHHQWKHVLJQDO:DQGWKHFRUUHFWFULWHULRQ'VKRXOGEHUHYLVLWHG7RDJUHDWH[WHQWVXEMHFWVHYHQZKHQVHDUFKLQJDUHVHDUFKLQJLQYDLQVLQFH WKHUH LVQR ORFDORSWLPXP WRHVFDSH IURP$QRQPRQRWRQLFIXQFWLRQZLWKORFDODQGJOREDORSWLPXPZRXOGSUREDEO\LQFUHDVHWKHH[SODQDWRU\SRZHURIWKHVWXG\)LQDOO\DFOHDUFHLOLQJHIIHFWFDQEHIRXQGUHVWULFWLQJRSWLRQVIRUIXUWKHUOHDUQLQJ,QWKLVVHQVHVXEMHFWVDUHDWULVNRIVXIIHULQJIURPWKHYLGHRJDPHHIIHFWGHULYHGIURPDQREVHUYDWLRQ WKDW FKLOGUHQ FDQ RIWHQ REWDLQ YHU\ KLJK VFRUHV SOD\LQJ YLGHRJDPHV EXWDVNHGKRZGLGWKH\GRLWRUZKDWRQHKDVWRGRLQRUGHUWRSOD\ZHOOKDYHGLIILFXOWLHVWRDQVZHU

3UDFWLFDO LPSOHPHQWDWLRQ $VH[SHFWHG LWKDVEHHQ VKRZQ WKDWXQGHU LPPHGLDWHIHHGEDFNLQFUHDVLQJWKHH[DFWLQJQHVVRIOHDUQLQJHQYLURQPHQWVLQGXFHVVWUDWHJ\VHDUFK%XWWKHFXUUHQWVWXG\JRHVRQHVWHSIXUWKHU$QRWKHUYDULDEOHLQGXFLQJVWUDWHJ\VHDUFKKDVEHHQLGHQWLILHGQDPHO\WKHIHHGEDFNVWLPLQJ,ILWLVWUXHWKDWGHOD\LQJWKHIHHGEDFNLQWKHH[DFWLQJHQYLURQPHQWFRXOGPDLQWDLQVXEMHFWVDGGLWLRQDOO\IRFXVHGRQPD[LPL]LQJWKHLUSHUIRUPDQFHDWVPDOOHVWSRVVLEOHGHYLDWLRQVIURPFXUUHQWO\XVHGVWUDWHJ\DQGWKDWGHOD\LQJ WKH IHHGEDFN LQ WKH OHQLHQW HQYLURQPHQW PDGH VXEMHFWV IRFXV HVSHFLDOO\ RQZLQQLQJVWUDWHJ\VHDUFKWKRVHIDFWVFDQKDYHDWUHPHQGRXVSUDFWLFDOLPSOHPHQWDWLRQV

7/29/2019 7-delayedfeedback

17/19

/HWXVLPDJLQHDIROORZLQJLOOXVWUDWLYHVHWXSD)LQDQFLDO$GYLVRU)$LVKHOSLQJVHYHUDOFOLHQWVWRPDQDJHWKHLUORQJWHUPLQYHVWPHQWVLQVWRFNV,IWKH)$DVDQH[SHUWEHOLHYHV WKDW WKH ILQDQFLDOPDUNHWV DUH SDVVLQJWKURXJK D WHPSRUDOGRZQWXUQ WKHUHIRUHZKDWHYHUDFWLRQDFOLHQWWDNHVHYHQDFRUUHFWRQHZLOOEULQJSHQDOL]LQJHIIHFWVH[DFWLQJHQYLURQPHQW LW VKRXOG EH MXVWLILHG WKDW VKH WKH )$ GHOD\V IHHGEDFN DERXW FOLHQWVSRUWIROLRVDQGVWRFNPDUNHWV SHUIRUPDQFH ,Q WKLVPDQQHU FOLHQWVEHDU VPDOOHU ULVN RIRYHUUHDFWLQJ LH VHOOLQJ JRRG VKDUHV DQG EX\LQJ DQRWKHU ZKLFK VHHP WR EHWHPSRUDULO\OHVVDIIHFWHGE\ WKH RYHUDOO VOXPS EXWDUH SULFHGDERYH WKHLU IXQGDPHQWDOYDOXH

2Q WKH RWKHU KDQG OHW XV WKLQN RI D FUHDWLYH DGYHUWLVLQJ FDPSDLJQ EHLQJGHYHORSHGIRUSURGXFW=6LQFHWKHSURGXFWLVQHZWRWKHPDUNHWDQGWKHSULRUPDUNHWVWXGLHVVKRZHGWKDWLWZRXOGVHOOZHOOZHPLJKWVD\WKDWWKHUHH[LVWVDELJSUREDELOLW\WKDWWKHSHQDOW\LQWHUPVRIVDOHVIRUQRWDGYHUWLVLQJWKHSURGXFWLQWKH RSWLPDOPDQQHULVJRLQJWREHVPDOOOHQLHQWHQYLURQPHQW7KHUHIRUHLWLVLQWKHLQWHUHVWRIWKHSURGXFHURI

SURGXFW=WRGHOD\WKHIHHGEDFNLQWHUPVRINQRZOHGJHRIVDOHVUDWHWRWKHPDUNHWLQJDJHQF\VRWRDYRLGEHLQJVWXFNLQWKHVXERSWLPDODGYHUWLVLQJFDPSDLJQ

(1':25'7KHVWXG\WKDWKDVEHHQGHVFULEHGLQWKLVSDSHUPDNHVWZRLPSRUWDQWFRQWULEXWLRQV)LUVWRIDOOLWDGGVWRDOUHDG\H[LVWLQJHYLGHQFHDERXWWKHLPSRUWDQWUROHRIGHOD\HGIHHGEDFN,WGHPRQVWUDWHV WKDW VRPH RI WKH SUHYLRXV EHOLHIV DERXW IHHGEDFN EHLQJ XVHIXO RQO\LPPHGLDWHO\DIWHU WKH GHFLVLRQ VKRXOGEH UHYLVLWHG 6HFRQGO\ WKLV SDSHURSHQV D QHZLPSRUWDQW DUHD IRU UHVHDUFK )HHGEDFNV WLPLQJ KDV VR IDU QRW EHHQ WDNHQ LQWR

FRQVLGHUDWLRQDVDYDULDEOHLQGXFLQJRUSUHVHUYLQJIURPWKHVWUDWHJ\FKDQJH7KHHYLGHQFHFROOHFWHGLQWKLVVWXG\VXJJHVWVWKDWWKLVPDWWHUVKRXOGEHLQYHVWLJDWHGIXUWKHU

7/29/2019 7-delayedfeedback

18/19

5HIHUHQFHV

5 / %DQJHUW'URZQV & & .XOLN - $ .XOLN DQG 0 0RUJDQ 7KH

LQVWUXFWLRQDO HIIHFW RI IHHGEDFN LQ WHVWOLNH HYHQWV 5HYLHZ RI (GXFDWLRQDO5HVHDUFK

:+%HUTXLVW DQG6 5 3KLOLSV$ KDQGERRN IRU IDFXOW\ GHYHORSPHQWYRO'DQVYLOOH1

7/29/2019 7-delayedfeedback

19/19

:70DGGR[)*$VKE\DQG&-%RKLO'HOD\HG IHHGEDFN HIIHFWVRQUXOHEDVHG DQG LQIRUPDWLRQLQWHJUDWLRQ FDWHJRU\ OHDUQLQJ >2QOLQH@ $YDLODEOHKWWSORYHSV\XWH[DVHGXORYHFRQFHSWV0DGGR[HWDOSGI

%-0DVRQDQG5%UXQLQJ3URYLGLQJ IHHGEDFNLQFRPSXWHUEDVHG LQVWUXFWLRQ:KDW WKH UHVHDUFK WHOOV XV >2QOLQH@ $YDLODEOHKWWSGZEXQOHGX(GLW0%0DVRQ%UXQLQJKWPO

* 5 0RUULVRQ 6 0 5RVV 0 *RSDODNULVKQDQ DQG - &DVH\7KH HIIHFWV RIIHHGEDFN DQG LQFHQWLYHV RQ DFKLHYHPHQW LQ FRPSXWHUEDVHG LQVWUXFWLRQ&RQWHPSRUDU\(GXFDWLRQDO3V\FKRORJ\QR

( + 0RU\ 7KH KDQGERRN RI UHVHDUFK IRU HGXFDWLRQDO FRPPXQLFDWLRQV DQGWHFKQRORJ\FK)HHGEDFN5HVHDUFKSS1HZ