Hadoop 2 @ Twitter, Elephant Scale

29
+DGRRS #7ZLWWHU (OHSKDQW 6FDOH /RKLW 9LMD\D5HQX *HUD 6KHJDORY #ORKLWYLMD\DUHQX #JHUDVKHJDORY #7ZLWWHU+DGRRS Y

description

Hadoop Summit 2014

Transcript of Hadoop 2 @ Twitter, Elephant Scale

Page 1: Hadoop 2 @ Twitter, Elephant Scale

+DGRRS���#7ZLWWHU��(OHSKDQW�6FDOH

/RKLW�9LMD\D5HQX�����������*HUD�6KHJDORY#ORKLWYLMD\DUHQX���������������������������#JHUDVKHJDORY

#7ZLWWHU+DGRRS���������Y�����������

Page 2: Hadoop 2 @ Twitter, Elephant Scale

$ERXW�WKLV�WDON

6KDUH�#WZLWWHUKDGRRS¶V�HIIRUWV��H[SHULHQFH�DQG�OHDUQLQJ�LQ�PRYLQJ�WKRXVDQG�XVHUV�DQG�PXOWL�SHWDE\WH�ZRUNORDGV�IURP�+DGRRS���WR�+DGRRS��

�������#WZLWWHUKDGRRS���������Y�����������

Page 3: Hadoop 2 @ Twitter, Elephant Scale

8VH�FDVHV�3HUVRQDOL]DWLRQ

*UDSK�DQDO\VLV��5HFRPPHQGDWLRQV��7UHQGV��8VHU�WRSLF�PRGHOLQJ$QDO\WLFV

D�E�WHVWLQJ��XVHU�EHKDYLRU�DQDO\VLV��DSL�DQDO\WLFV*URZWK

1HWZRUN�'LJHVW��3HRSOH�5HFRPPHQGDWLRQV��(PDLO5HYHQXH

(QJDJHPHQW�SUHGLFWLRQ��$G�WDUJHWLQJ��DGV�DQDO\WLFV��PDUNHWSODFH�RSWLPL]DWLRQ1LHOVHQ�7ZLWWHU�79�5DWLQJ

7ZHHW�LPSUHVVLRQV�SURFHVVLQJ%DFNXSV��6FULEH�/RJV

0\64/�EDFNXSV��0DQKDWWDQ�EDFNXSV��)URQW(QG�VFULEH�ORJV0DQ\�PRUH���

�������#WZLWWHUKDGRRS���������Y�����������

Page 4: Hadoop 2 @ Twitter, Elephant Scale

+DGRRS�DQG�'DWD�SLSHOLQH

7)( KDGRRS�UHDO�WLPH

KDGRRS�SURFHVVLQJ

KDGRRS�ZDUHKRXVH

KDGRRS�FROG

KDGRRS�EDFNXSV6HDUFK��

$GV��HWF 3DUWQHUV

0\64/

KDGRRSKEDVH

9HUWLFD 0DQKDWWDQ

KDGRRSWVW

�������#WZLWWHUKDGRRS

691��*LW�����

KDGRRSWVW

���������Y�����������

Page 5: Hadoop 2 @ Twitter, Elephant Scale

(OHSKDQW�6FDOH

䚉 7HQV�RI�WKRXVDQGV�+DGRRS�VHUYHUV��0L[�RI�KDUGZDUH�

䚉 +XQGUHGV�RI�WKRXVDQGV�RI�GLVN�GULYHV䚉 )HZ�KXQGUHG�3%�GDWD�VWRUHG�LQ�+')6䚉 +XQGUHGV�RI�WKRXVDQGV�RI�GDLO\�

KDGRRS�MREV䚉 7HQV�RI�PLOOLRQV�RI�GDLO\�KDGRRS�WDVNV

�������#WZLWWHUKDGRRS

,QGLYLGXDO�&OXVWHU�6WDWV

䚉 0RUH�WKDQ������QRGHV䚉 �������3%�GDWD�VWRUHG�LQ�+')6䚉 ��.�53&�VHFRQG�RQ�11V䚉 ��.��MREV�SHU�GD\䚉 ��0��WDVNV�SHU�GD\䚉 �3%��GDWD�FUXQFKHG�SHU�GD\

���������Y�����������

Page 6: Hadoop 2 @ Twitter, Elephant Scale

+DGRRS���&KDOOHQJHV��4�������*URZWK�

6XSSRUWLQJ�WZLWWHU�JURZWK��5HTXHVW�IRU�QHZ�IHDWXUHV�RQ�ROGHU�EUDQFK��QHZ�-$9$

6FDODELOLW\��1DPH1RGH�ILOHV�EORFNV��11�2SHUDWLRQV��*&�SDXVH��&KHFNSRLQWLQJ-RE7UDFNHU�*&�SDXVH��WDVN�DVVLJQPHQW

5HOLDELOLW\�632)�11�DQG�-7��1DPH1RGH�UHVWDUW�GHOD\V

(IILFLHQF\�6ORW�XWLOL]DWLRQ��4R6��0XOWL�7HQDQW��1HZ�IHDWXUHV��IUDPHZRUNV�

0DLQWHQDQFH��2OG�FRGHEDVH��1XPHURXV�LVVXHV�IL[HG�LQ�ODWHU�YHUVLRQV��GHY�EUDQFK

�������#WZLWWHUKDGRRS���������Y�����������

Page 7: Hadoop 2 @ Twitter, Elephant Scale

+DGRRS���&RQILJXUDWLRQ��4�������

1RGH0DQDJHU'DWD1RGH

1RGH0DQDJHU'DWD1RGH

1RGH0DQDJHU'DWD1RGH

<$51�5HVRXUFH0DQDJHU

�-1 �-1 �-1 �-1 �-1 �-1

9LHZ)6��+')6�%DODQFHU��$GPLQ�WRROV��K5DYHQ��0HWULFV�$OHUWV

««� ««�

ORJV XVHU WPS7UDVK

�������#WZLWWHUKDGRRS

7UDVK7UDVK

���������Y�����������

Page 8: Hadoop 2 @ Twitter, Elephant Scale

+DGRRS���0LJUDWLRQ��4��4�������

3KDVH�����7HVWLQJ

3KDVH�����3URGXFWLRQ�

3KDVH�����6HPL�SURGXFWLRQ

䚉 $SDFKH�������EUDQFK䚉 1HZ�+DUGZDUH ��1HZ�

26�DQG�-90䚉 %HQFKPDUNV�DQG�XVHU�

MREV��ORWV�RI�WKHP«�䚉 'HSHQGHQW�

FRPSRQHQW�XSGDWHV䚉 'DWD�PRYHPHQW�

EHWZHHQ�GLIIHUHQW�YHUVLRQV

䚉 0HWULFV��$OHUWV�DQG�WRROV䚉 3URGXFWLRQ�XVH�FDVHV�

UXQQLQJ�LQ���FOXVWHUV�LQ�SDUDOOHO�

䚉 7XQLQJ�SDUDPHWHU�XSGDWHV�DQG�OHDUQLQJV

䚉 6WDUWHG�FRQWULEXWLQJ�IL[HV�EDFN�WR�FRPPXQLW\

䚉 (GXFDWLQJ�XVHUV�DERXW�QHZ�YHUVLRQ�DQG�FKDQJHV

䚉 %HQHILWV�RI�+DGRRS��

䚉 6WDEOH�$SDFKH�������UHOHDVH�ZLWK�PDQ\�IL[HV�DQG�EDFNSRUWV

䚉 0XOWLSOH�LQWHUQDO�UHOHDVHV

䚉 7HPSODWH�IRU�QHZ�FOXVWHUV

䚉 5HDG\�WR�UROO�$SDFKH�����UHOHDVH�

KWWS���ZZZ�VOLGHVKDUH�QHW�+DGRRSB6XPPLW�KDGRRS�KDUGZDUH�WZLWWHU�VL]H�GRHV�PDWWHU�������#WZLWWHUKDGRRS

���������Y�����������

Page 9: Hadoop 2 @ Twitter, Elephant Scale

&38�8WLOL]DWLRQ�

+DGRRS���&38�8WLOL]DWLRQ�IRU�RQH�GD\�������SHDNV�

+DGRRS���&38�8WLOL]DWLRQ�IRU�RQH�GD\�������SHDNV�

�������#WZLWWHUKDGRRS���������Y�����������

Page 10: Hadoop 2 @ Twitter, Elephant Scale

0HPRU\�8WLOL]DWLRQ�

+DGRRS���0HPRU\�8WLOL]DWLRQ�IRU�RQH�GD\������SHDNV�

+DGRRS���0HPRU\�8WLOL]DWLRQ�IRU�RQH�GD\������SHDNV�

�������#WZLWWHUKDGRRS����������Y�����������

Page 11: Hadoop 2 @ Twitter, Elephant Scale

0LJUDWLRQ�&KDOOHQJH��ZHE�EDVHG�)6

1HHG�D�ZHE�EDVHG�)6�WR�GHDO�ZLWK�+��+��LQWHUDFWLRQVƔ +IWS�EDVHG�RQ�FURVV�'&�/RJ0RYHU�H[SHULHQFHƔ $SSV�EURNHQ�GXH�WR�QR�)1)�RQ�QRQ�H[LVWLQJ�SDWKV�

+')6�����Ɣ )DFHG�FKDOOHQJHV�FURVV�YHUVLRQ�FKHFNVXPV

�������#WZLWWHUKDGRRS����������Y�����������

Page 12: Hadoop 2 @ Twitter, Elephant Scale

0LJUDWLRQ�&KDOOHQJH��KDUG�FRGHG�)6

����¶V�RI�RFFXUUHQFHV�KGIV����^11`�SDWK�DQG�DEVROXWH�85,VƔ )RU�FOXVWHU��GLDO�KGIV���KDGRRS�FOXVWHU��QQ�GF�&1$0(Ɣ )RU�FOXVWHU��GLDO�«

,GHDO��XVH�ORJLFDO�SDWKV�DQG�YLHZIV�DV�GHIDXOW)6

0RUH�UHDOLVWLF�DQG�IDVWHU�Ɣ +')6&RPSDWLEOH9LHZ)6�+$'223�����

�������#WZLWWHUKDGRRS����������Y�����������

Page 13: Hadoop 2 @ Twitter, Elephant Scale

0LJUDWLRQ�&KDOOHQJH��,QWHURSHUDELOLW\

0LJUDWLRQ�LQ�SURJUHVV��+��MRE�UHTXLUHV�LQSXW�IURP�+�Ɣ KIWS���20*ZKDW11�KDV�P\�SDWK�SUREOHPƔ LGHDO��XVH�YLHZIV�RQ�+��UHVROYLQJ�WR�FRUUHFW�+��11Ɣ UHDOLVWLF��VHH�DERYH�³KDUGFRGHG�)6´Ɣ (YHQ�LI�\RX�NQRZ�20*ZKDW11��LV�LW�DFWLYH"

�������#WZLWWHUKDGRRS����������Y�����������

Page 14: Hadoop 2 @ Twitter, Elephant Scale

6WDQGE\$FWLYH&OXVWHU&1$0(

+��FOLHQW

$FWLYH 6WDQGE\ $FWLYH 6WDQGE\

/RDG�FOLHQW�VLGH�PRXQWWDEOH�RQ�WKH�VHUYHU�VLGH���� UHGLUHFW�WR�WKH�ULJKW�

QDPHVSDFH�� UHGLUHFW�WR�DFWLYH�ZLWKLQ�

QDPHVSDFH

�������#WZLWWHUKDGRRS����������Y�����������

Page 15: Hadoop 2 @ Twitter, Elephant Scale

0LJUDWLRQ��7RROV�DQG�(FRV\VWHP

Ɣ 3RUW�UHFRPSLOH�SDFNDJH��ż 'DWD�$FFHVV�/D\HU�+&DWDORJ��ż 3LJ��ż &DVFDGLQJ�6FDOGLQJż (OHSKDQW%LUGż KDGRRS�O]R

Ɣ 3,*�������ORFDO�PRGH�FRXQWHUV��Ɣ $QDO\WLFV�WHDP�IL[HG�3,*�������SHUIRUPDQFH�Ɣ K5DYHQ�IL[HV�

ż WUDQVODWLRQ�EHWZHHQ�VORWBPLOOLV�DQG�PEBPLOOLV

�������#WZLWWHUKDGRRS����������Y�����������

Page 16: Hadoop 2 @ Twitter, Elephant Scale

+DG2RSV�IRXQG�DQG�IL[HG

Ɣ 9LHZ)6�FDQ¶W�EH�XVHG�IRU�SXEOLF�'LVWULEXWHG&DFKH��'&�ż +$'223��������<$51�����

Ɣ JHW)LOH6WDWXV�53&�VWRUP�RQ�SXEOLF�'&��ż <$51�����

Ɣ 1R�XVHU�VSHFLILHG�SURJUHVV�VWULQJ�LQ�05�$0�8,�WDVNż 0$35('8&(�����

Ɣ 8EHUL]HG�MREV�IRU�VFKHGXOLQJ�VPDOO�MREV�JUHDW�EXW�����ż FDQ�\RX�NLOO�WKHP"�0$35('8&(�����ż VL]H�FRUUHFWO\�IRU�PDS�RQO\"�<$51�����

�������#WZLWWHUKDGRRS����������Y�����������

Page 17: Hadoop 2 @ Twitter, Elephant Scale

0RUH�+DG2RSV

,QFLGHQW��D�MRE�EODFNOLVWV�QRGHV�E\�ORJJLQJ�WHUDE\WHVƔ QHHG�FDSSLQJ��EXW�XVHUORJ�OLPLW�NE�ORVHV�YDOXDEOH�ORJ�WDLOƔ 5ROOLQJ)LOH$SSHQGHU�IRU�05�$0�WDVNV�0$35('8&(�

����

�������#WZLWWHUKDGRRS����������Y�����������

Page 18: Hadoop 2 @ Twitter, Elephant Scale

'LDJQRVWLFV�LPSURYHPHQW

$SS�-RE�7DVN�NLOO�Ɣ '$*�SURFHVVRUV�XVHUV�FDQ�VD\�ZK\

ż 0$35('8&(�������<$51�����

Ɣ 05�$0��³VSHFXODWLRQ´��³UHGXFHU�SUHHPSWLRQ´ż 0$35('8&(�������0$35('8&(�����

Ɣ 7KUHDG�'XPSVż 2Q�WDVN�WLPHRXW��0$35('8&(�����ż 2Q�GHPDQG�IURP�&/,�8,��0$35('8&(������������

�������#WZLWWHUKDGRRS����������Y�����������

Page 19: Hadoop 2 @ Twitter, Elephant Scale

8;�8,�LPSURYHPHQWV

Ɣ 1DPH1RGH�VWDWH�DQG�FOXVWHU�VWDWVƔ $SS�VL]H�LQ�0%�RQ�50�$SSV�3DJHƔ 50�6FKHGXOHU�8,�LPSURYHPHQWV��TXHXH�GHVFULSWLRQV��

EXJV�PLQ�PD[�UHVRXUFH�FDOF�Ɣ 7DVN�$WWHPSW�VWDWH�ILOWHULQJ�LQ�05�$0

+')6�������<$51�������+')6��������

�������#WZLWWHUKDGRRS����������Y�����������

Page 20: Hadoop 2 @ Twitter, Elephant Scale

<$51�UHOLDELOLW\�LPSURYHPHQWV

Ɣ 8QKHDOWK\�QRGHV���SRVLWLYH�IHHGEDFNż GUDLQ�FRQWDLQHUV�LQVWHDG�RI�NLOOLQJ��<$51������ż GRQ¶W�UHUXQ�PDSV�ZKHQ�DOO�UHGXFHV�FRPPLWWHG��0$35('8&(�����

�Ɣ 50�FUDVKHV�-,5$�IL[HG�HLWKHU�MXVW�LQWHUQDOO\�RU�SXEOLF

ż <$51������<$51����

�������#WZLWWHUKDGRRS����������Y�����������

Page 21: Hadoop 2 @ Twitter, Elephant Scale

0DS5HGXFH�XVDELOLW\

Ɣ 0HPRU\�PE�DV�D�VLQJOH�WXQDEOH��;P[��VRUW�PE�DXWR�VHWż PE�LV�RSWLPL]HG�RQ�FDVH�E\�FDVH�EDVLVż 0$35('8&(�����

Ɣ 8VHUV�ZDQW�QHZHU�DUWLIDFWV�OLNH�JXDYD��MRE�FODVVORDGHUż 0$35('8&(��������������������������

Ɣ +HOS�XVHUV�GHEXJż WKUHDG�GXPS�RQ�WLPHRXW��DQG�RQ�GHPDQG�YLD�8,ż HGXFDWH�XVHUV�DERXW�KHDS�GXPSV�RQ�220�DQG�MDYD�SURILOLQJ

�������#WZLWWHUKDGRRS����������Y�����������

Page 22: Hadoop 2 @ Twitter, Elephant Scale

0XOWL�'&�HQYLURQPHQW

05�FOLHQWV�DFURVV�ODWHQF\�ERXQGDULHV��6XEPLW�IDVW��Ɣ PRYLQJ�VSOLW�FDOFXODWLRQ�WR�05�$0��0$35('8&(����

'6&3�ELW�FRORULQJ�IRU�'DWD;IHUƔ +')6�����Ɣ +IWS��VZLWFKHG�WR�$SDFKH�&RPPRQV�+WWS&OLHQW�

'DWD;IHU�WKURWWOLQJ��FOLHQW�5:�

����������Y�����������

Page 23: Hadoop 2 @ Twitter, Elephant Scale

<$51��%H\RQG�-DYD��0DS5HGXFH

Ɣ 05�$0�DQG�RWKHU�5(67�$3,¶V�DFURVV�WKH�VWDFN�IRU�HDV\�LQWHJUDWLRQ�LQ�QRQ�-90�WRROV�

Ɣ 9RZSDO�:DEELW���SURGXFWLRQ�ż QR�H[WUD�VSDQQLQJ�WUHH�VWHS�

Ɣ 6SDUN��VHPL�SURGXFWLRQ�

�������#WZLWWHUKDGRRS����������Y�����������

Page 24: Hadoop 2 @ Twitter, Elephant Scale

2QJRLQJ�3URMHFW��6KDUHG�&DFKH

0DS5HGXFH�IXQFWLRQ�VKLSSLQJ��FRPSXWDWLRQ�!GDWDƔ 7HDPV�KDYH�MREV�ZLWK����¶V�RI�MDUV�XSORDGHG�YLD�OLEMDUV

ż ,GHDO��PDQDJH�D�MDU�UHSR�RQ�+')6ż 5HIHUHQFH�MDUV�YLD�'LVWULEXWHG&DFKH�LQVWHDG�RI�XSORDGLQJż 5HDO��FXUUHQWO\�KDUG�WR�FRRUGLQDWH

�Ɣ <$51�������0DQDJH�DUWLIDFWV�FDFKH�WUDQVSDUHQWO\

Ɣ 0HDVXUH�LW�ż <$51�������/RFDOL]DWLRQ�RYHUKHDG�FDFKH�KLWV�10�PHWULFVż 0$35('8&(��������-RE�ORFDOL]DWLRQ�FRXQWHUV

�������#WZLWWHUKDGRRS����������Y�����������

Page 25: Hadoop 2 @ Twitter, Elephant Scale

8SFRPLQJ�&KDOOHQJHV

Ɣ 5HGXFH�RSV�FRPSOH[LW\��ż JURZ�WR���.��QRGH�FOXVWHUVż WU\�WR�DYRLG�DGGLQJ�PRUH�FOXVWHUV�

Ɣ 6FDODELOLW\�OLPLWV�IRU�11��50Ɣ 11�KHDS�VL]HV��ODUJH�-DYD�KHDS�YV�QDPHVSDFH�VSOLWWLQJƔ 53&�4R6�,VVXHVƔ 11�VWDUWXS��ORQJ�LQLWLDO�EORFN�UHSRUW�SURFHVVLQJƔ ,QWHJUDWLQJ�QRQ�05�IUDPHZRUNV�ZLWK�K5DYHQ

�������#WZLWWHUKDGRRS����������Y�����������

Page 26: Hadoop 2 @ Twitter, Elephant Scale

)XWXUH�:RUN�,GHDV

Ɣ 3URGXFWL]H�50�+$�DQG�ZRUN�SUHVHUYLQJ�UHVWDUW�Ɣ +')6�5HDGDEOH�6WDQGE\�11Ɣ :KROH�'$*�LQ�D�VLQJOH�11�QDPHVSDFHƔ &RQWULEXWH�WR�+')6��������'HGLFDWHG�%0�VHUYLFHƔ 11�6/$��IDLUVKDUH�IRU�53&�TXHXHV��+$'223������Ɣ )LQHU�ORFN�JUDQXODULW\�LQ�11

�������#WZLWWHUKDGRRS����������Y�����������

Page 27: Hadoop 2 @ Twitter, Elephant Scale

6XPPDU\��+DGRRS���#�7ZLWWHU

Ɣ 1R�-7�ERWWOHQHFN��/LJKWZHLJKW�50���05�$0Ɣ +LJK�FRPSXWH�GHQVLW\�ZLWK�IOH[LEOH�VORWVƔ 5HGXFHG�11�ERWWOHQHFN�XVLQJ�)HGHUDWLRQƔ +')6�+$�UHPRYHV�WKH�DQJVW�WR�WU\�RXW�QHZ�11�FRQILJVƔ 0XFK�FORVHU�WR�XSVWUHDP�WR�FRQVXPH�FRQWULEXWH�IL[HV

ż 'HYHORSPHQW�RQ�����EUDQFKƔ $GRSWLQJ�QHZ�IUDPHZRUNV�RQ�<$51

�������#WZLWWHUKDGRRS����������Y�����������

Page 28: Hadoop 2 @ Twitter, Elephant Scale

&RQFOXVLRQ

0LJUDWLQJ�������XVHUV�XVH�FDVHV�LV�DQ\WKLQJ�EXW�WULYLDO«�KRZHYHU�

Ɣ +DGRRS���PDGH�LW�ZRUWKZKLOHƔ +DGRRS���FRQWULEXWLRQV��

ż ����SDWFKHV�FRPPLWWHGż a���LQ�UHYLHZ

�������#WZLWWHUKDGRRS����������Y�����������

Page 29: Hadoop 2 @ Twitter, Elephant Scale

7KDQN�\RX��4XHVWLRQV�

#-RLQ7KH)ORFN�DERXW�WZLWWHU�FRP�FDUHHUV

#7ZLWWHU+DGRRS

&DWFK�XS�ZLWK�XV�LQ�SHUVRQ#/RKLW9LMD\D5HQX��������������������������#*HUD6KHJDORY

�������#WZLWWHUKDGRRS����������Y�����������