Productionizing a 24/7 Spark Streaming Service on YARN

29
3URGXFWLRQL]LQJ D 6SDUN 6WUHDPLQJ 6HUYLFH RQ <$51 ,VVDF %XHQURVWUR $UXS 0DODNDU 6SDUN 6XPPLW -XO\

Transcript of Productionizing a 24/7 Spark Streaming Service on YARN

3URGXFWLRQL]LQJ�D������6SDUN�6WUHDPLQJ�6HUYLFH�RQ�<$51

,VVDF�%XHQURVWUR��$UXS�0DODNDU

6SDUN�6XPPLW�����-XO\��������

$ERXW�2R\DOD

)RXQGHG�LQ�����

�����HPSOR\HHV�ZRUOGZLGH

*OREDO�IRRWSULQW�RI����0�XQLTXH�XVHUV�LQ�����FRXQWULHV

2R\DOD�ZRUNV�ZLWK�WKH�PRVW�VXFFHVVIXO�EURDGFDVWV�DQG�PHGLD�FRPSDQLHV�LQ�WKH�ZRUOG

5HDFK��PHDVXUH��PRQHWL]H�YLGHR�EXVLQHVV

&URVV�GHYLFH�YLGHR�DQDO\WLFV�DQG�PRQHWL]DWLRQ�SURGXFWV�DQG�VHUYLFHV

2R\DOD�$QDO\WLFV�(7/

�� ,QJHVW�DQG�SURFHVV��%��HYHQWV�GD\�� 'HFRGH��!�(QULFK��!�3HUVLVW

5XQ����������PLQ�ODWHQF\������GDWD�ORVV��UHGXFH�&2*6��PLQLPDO�KXPDQ�LQWHUDFWLRQ�

3OD\V�'LVSOD\V��&RQYHUVLRQ�5DWH��*HR��'HYLFH��26��8QLTXH��(QJDJHPHQW��HWF�

$UFKLWHFWXUH�GLDJUDP

+773�5HFHLYHU .$).$

6SDUN�6WUHDPLQJ�RQ�<$51

+')6

7DNH�+RPH�0HVVDJH<HV��WKHUH�DUH�LVVXHV�ZLWK�6SDUN�6WUHDPLQJ

+RZHYHU�

Ɣ $OORZV�IOH[LELOLW\�QRW�DYDLODEOH�LQ�RWKHU�IUDPHZRUNV

Ɣ ,VVXHV�DUH�QRW�KXJH��ZRUNLQJ�RQ�UHVROYLQJ�WKHP

Ɣ $ZDUHQHVV���EHVW�SUDFWLFHV

+\EULG�3URFHVVLQJ

%DWFKƔ (DV\�UHSURFHVVLQJƔ 0RUH�KLVWRULFDO�

DZDUHQHVVƔ &RQVLVWHQF\�

JXDUDQWHHV

6WUHDPLQJƔ 5HDO�WLPHƔ /RZHU�VFKHGXOLQJ�

QHHGV

:K\�6SDUN"

2QH�FRGHEDVH�WR�PDLQWDLQ�DQG�WHVW

6DPH�IUDPHZRUN�IRU�%DWFK�DQG�6WUHDPLQJ

(DVLO\�PDLQWDLQDEOH�FOXVWHU���YV�6WRUP��

2XW�RI�WKH�ER[�<$51�LQWHJUDWLRQ

:K\�<$51"Ɣ (OLPLQDWH�RSHUDWLRQDO�FRVW�RI�PDLQWDLQLQJ�D�

VHSDUDWH�VSDUN�FOXVWHUƔ %HWWHU�+�:�XWLOL]DWLRQƔ (DV\�XSJUDGH

ż ����WR�������XSJUDGH�MXVW�D�GHSHQGHQF\�FKDQJH�ż YV�XSJUDGLQJ�D�GHGLFDWHG�FOXVWHU

Ɣ /HYHUDJH�WKH�H[SHUWLVH�GHDOLQJ�ZLWK�<$51�KDGRRS

<$51�3LWIDOOV

Ɣ ����0%�RYHUKHDG�SHU�FRQWDLQHU

Ɣ 'LVFRQQHFW�EHWZHHQ�6SDUN�<$51�8,

Ɣ <$51�FODVVSDWK�DXWRPDWLFDOO\�DGGHG�WR�6SDUN

2XWOLQH�� /DXQFKLQJ�VWUHDPLQJ�DSSOLFDWLRQV

D� �����UXQQLQJ�� ,QVWUXPHQWDWLRQ

D� WUDFNLQJ�RI�ZRUNHUV�� &RQVXPLQJ�VWUHDPLQJ�GDWD

D� IL[LQJ�FRQVXPSWLRQ�VSHHG�UHOLDELOLW\�LVVXHV�� /RJJLQJ

D� 5HGXFLQJ�DPRXQW�RI�ORJV�� 3HUVLVWLQJ�VWUHDPLQJ�GDWD

/DXQFKLQJ�LQ�<$517UDGLWLRQDO�ODXQFKLQJ��6SDUN��������

H[SRUW�63$5.B-$5 �XVU�OLE�VSDUN�DVVHPEO\�������61$36+27�KDGRRS������MDU

MDYD��FS��HWF�KDGRRS�FRQI�$SS-DU�MDU�VSDUN�DVVHPEO\�MDU�RUJ�DSDFKH�VSDUN�GHSOR\�\DUQ�&OLHQW���MDU�$SS-DU�MDU���DGG-DUV��MDUV�FRQILJ�MDU���FODVV�RR\DOD�DSS�0DLQ&ODVV���DUJ�DUJ����DUJ�DUJ����QDPH�0\$SS

Ɣ /DXQFKHV�DSSOLFDWLRQ�LQ�<$51�Ɣ 7UDFNV�DSSOLFDWLRQ�XQWLO�GRQH��QRW�JUHDW�IRU�VWUHDPLQJ�

/DXQFKLQJ���6WUHDPLQJ�ZD\6WUHDPLQJ�ODXQFKLQJ�

MDYD��FS��HWF�KDGRRS�FRQI�$SS-DU�MDU�VSDUN�DVVHPEO\�MDU�RUJ�DSDFKH�VSDUN�\DUQ�6WUHDPLQJ&OLHQW���MDU�$SS-DU�MDU���DGG-DUV��MDUV�FRQILJ�MDU���FODVV�RR\DOD�DSS�0DLQ&ODVV���DUJ�DUJ����DUJ�DUJ����QDPH�0\$SS

Ɣ /DXQFK�DSSOLFDWLRQ�,)�QRW�UXQQLQJƔ &RQQHFW�WR�ROG�DSSOLFDWLRQ�LI�DOUHDG\�H[LVWVƔ 7UDFNV�DSSOLFDWLRQ�XQWLO�GRQHƔ 'RHV�QRW�NLOO�DSSOLFDWLRQ�LI�FUDVKHG

/DXQFKLQJ���5HODXQFKLQJ

8VH�XSVWDUW�ZLWK�6WUHDPLQJ�ODXQFKHU

Ɣ /DXQFK�DSSOLFDWLRQ�IURP�UHPRWH�KRVWƔ ,I�ODXQFKHU�IDLOV��DSSOLFDWLRQ�XQDIIHFWHGƔ 2Q�ODXQFKHU�VWDUW��FRQQHFW�WR�H[LVWLQJ�DSSƔ ,I�DSSOLFDWLRQ�IDLOV��XSVWUHDP�UH�ODXQFKHV�LW

)DLOXUH�&DVHV

3URFHVVLQJ�VWRSV��FRQVXPLQJ�FRQWLQXHV

%URDGFDVW�YDULDEOHV�JHW�*&HG��63$5.������

&RQVXPHUV�IDLO�VLOHQWO\��63$5.������

2XW�RI�0HPRU\�LI�FRQVXPLQJ�WRR�IDVW��63$5.�

�����

,QVWUXPHQWDWLRQ���&XVWRP�6RXUFHV

&RGDKDOH�0HWULFV

YDO�VRXUFH� �QHZ�6RXUFH�^����RYHUULGH�GHI�VRXUFH1DPH� ��RR\DOD�H[DPSOH�PHWULFV�����RYHUULGH�GHI�PHWULF5HJLVWU\� �QHZ�0HWULF5HJLVWU\����YDO�QXP5HT0HWHU� �PHWULF5HJLVWU\�PHWHU��QXP5HTXHVWV��`6SDUN(QY�JHW�PHWULFV6\VWHP�UHJLVWHU6RXUFH�VRXUFH�VRXUFH�QXP5HT0HWHU�PDUN��

,QVWUXPHQWDWLRQ���&UHDWH�DQ�LQVWUXPHQWDWLRQ

YDO�LQVWUXPHQWDWLRQ� �QHZ�6SDUN,QVWUXPHQWDWLRQ��RR\DOD�H[DPSOHV�PHWULFV��

���&UHDWH�DQ�DFFXPXODWRU

YDO�QXP5HTV� �VF�DFFXPXODWRU��/�

���5HJLVWHU�DFFXPXODWRU�LQ�WKH�LQVWUXPHQWDWLRQ

LQVWUXPHQWDWLRQ�VRXUFH�UHJLVWHU'DLO\$FFXPXODWRU�QXP5HTV���QXP5HTV��

���5HJLVWHU�WKH�LQVWUXPHQWDWLRQ�ZLWK�6SDUN

���0XVW�EH�GRQH�DIWHU�DOO�DFFXPXODWRUV�KDYH�DOUHDG\�EHHQ�UHJLVWHUHG

LQVWUXPHQWDWLRQ�UHJLVWHU��

,QVWUXPHQWDWLRQ���&XVWRP�6LQNVFODVV�'DWDGRJ6LQN�YDO�SURSHUW\��3URSHUWLHV��YDO�UHJLVWU\��0HWULF5HJLVWU\��H[WHQGV�6LQN�^��YDO�''B$3,B.(<� ��DSLNH\���YDO�DSLNH\� �SURSHUW\7R2SWLRQ�''B$3,B.(<��JHW

��YDO�UHSRUWHU� �'DWDGRJ5HSRUWHU�IRU5HJLVWU\�UHJLVWU\������FRQYHUW5DWHV7R�7LPH8QLW�6(&21'6������FRQYHUW'XUDWLRQV7R�7LPH8QLW�0,//,6(&21'6������EXLOG�WUDQVSRUW��KRVW�

��RYHUULGH�GHI�VWDUW���^�UHSRUWHU�VWDUW�SROO3HULRG��SROO8QLW��`��RYHUULGH�GHI�VWRS���^�UHSRUWHU�VWRS���``

,QVWUXPHQWDWLRQ���([DPSOH

&RQVXPLQJ�'DWD���'RFV�:D\

YDO�FRQI� �QHZ�6SDUN&RQI���VHW$SS1DPH�QDPH��VHW0DVWHU�VSDUN0DVWHU�

YDO�VF� �QHZ�6SDUN&RQWH[W�FRQI�YDO�VVF� �QHZ�6WUHDPLQJ&RQWH[W�VF��6HFRQGV�����

YDO�G6WUHDP� ��.DIND8WLOV�FUHDWH6WUHDP�VVF��]N&RQQHFW��FRQVXPHU*URXS��0DS�WRSLF��!������6WRUDJH/HYHO�0(025<B21/<��PDS��[� !�[�B��

&RQVXPLQJ�'DWD���,VVXHV

$OO�FRQVXPHUV�LQ�VDPH�QRGH

&RQVXPSWLRQ�FRXOG�EH�WRR�IDVW

&RQVXPSWLRQ�LQGHSHQGHQW�RI�SURFHVVLQJ

��������5HFHLYHU7UDFNHU�VWXFN��63$5.������

&RQVXPLQJ�'DWD���)L[HVYDO�FRQI� �QHZ�6SDUN&RQI���VHW$SS1DPH�QDPH��VHW0DVWHU�VSDUN0DVWHU����VHW��VSDUN�VWUHDPLQJ�UHFHLYHU�PD[5DWH���PD[5DWH�WR6WULQJ�����WKURWWOLQJ

YDO�VF� �QHZ�6SDUN&RQWH[W�FRQI�YDO�VVF� �QHZ�6WUHDPLQJ&RQWH[W�VF��6HFRQGV�����

YDO�NDIND,QSXWV� ����WR�QXP3DUWLWLRQV��PDS�^�B� !��.DIND8WLOV�FUHDWH6WUHDP�VVF��]N&RQQHFW��FRQVXPHU*URXS��0DS�WRSLF��!��������6WRUDJH/HYHO�0(025<B21/<��PDS��[� !�[�B��`YDO�G6WUHDP� �VVF�XQLRQ�NDIND,QSXWV�

&RQVXPLQJ�'DWD���'DWD�/RVV

6WUHDPLQJ�GDWD�UHSOLFDWHG�DFURVV�QRGHVƔ 8S�WR�RQH�EDWFK�ORVW�RQ�FUDVKƔ 'DWD�ORVV�LI�WDVNV�KDQJƔ .DIND�DXWRFRPPLW�PD\�SUHYHQW�UHFRYHU\

'LVDEOH�DXWRFRPPLW"�.HHS�WUDFN�RI�ILQLVKHG�RIIVHWV"

/RJJLQJ���:KHUH�WR�)LQGƔ <$51�PDNHV�ORJ�DYDLODEOH�LQ�+')6�IRU�ROG�MREV

Ɣ /RJV�RI�UXQQLQJ�MREV�DUH�DYDLODEOH�LQ�WKH�<$51�8,�

Ɣ 6SDUN�VKRXOG�OLQN�WR�KDGRRS�ORJ�8,�IURP�63$5.�8,

Ɣ 63$5.B/2*�-B&21)�PDNHV�LW�HDV\�WR�VHW�FXVWRP�ORJ�M�FRQILJXUDWLRQ�ILOH��PD\EH�FRQIOLFWV�ZLWK�<$51�FS��63$5.������

/RJJLQJ���5HGXFHG�/RJVƔ 7RR�PXFK�ORJJLQJ� !�VORZ�DSSOLFDWLRQ��ORJV�KDUG�WR�LQWHUSUHWƔ 6HW�ORJ�OHYHO�WR�:$51�IRU�FHUWDLQ�VSDUN�SDFNDJHV��VHW�

DGGLWLYLW\�WR�IDOVHż ORJ�M�ORJJHU�RUJ�DSDFKH�VSDUN�UGG�1HZ+DGRRS5'' :$51��

�DSSHQGHU!ż ORJ�M�DGGLWLYLW\�RUJ�DSDFKH�VSDUN�UGG�1HZ+DGRRS5'' IDOVH

Ɣ 5HFRPPHQGHG�IRU��ż ORJ�M�ORJJHU�RUJ�DSDFKH�VSDUN�^VWRUDJH��VFKHGXOHU��

&DFKH7UDFNHU��&DFKH7UDFNHU$FWRU��0DS2XWSXW7UDFNHU$FWRU��0DS2XWSXW7UDFNHU��H[HFXWRU`

6WUHDPLQJ�2XWSXWV

$WRPLF�ZULWLQJ�HQVXUHV�FRQVLVWHQF\��WHPS���PRYH�

)LQH�FRQWURO�RI�RXWSXW�XVLQJ�IRU(DFK5GG��PDS3DUWLWLRQV

&RPSOHWLRQ,WHUDWRU�SURYLGHV�ILQHU�FRQWURO�RQ�ILOH�FORVLQJ

H�J��2R\DOD�(7/�XVHV�FROXPQDU�SDUTXHW�ILOHV�LQ�+')6

&RQFOXVLRQ

Ɣ 5HDO�WLPH�SLSHOLQH�UHTXLUHG�IRU�ORZ�ODWHQF\

Ɣ 0HVVDJH�GURSV�KDUG�WR�DYRLG���UHSURFHVVLQJ�LV�KDUG

Ɣ +\EULG�PRGHO�RI�UHDO�WLPH�IROORZHG�E\�EDWFK�SURFHVVLQJ�JLYHV�EHVW�RI�ERWK�ZRUOGV

Ɣ ,QVWUXPHQWDWLRQ�LV�NH\�WR�D�KHDOWK\�UHDO�WLPH�SLSHOLQH

Ɣ /RJJLQJ�QHHGV�PDMRU�IL[LQJ

48(67,216,VVDF�%XHQURVWUR���EXHQURVWUR#RR\DOD�FRP

$UXS�0DODNDU���DUXS#RR\DOD�FRP

&RGH�VDPSOHV�DW�KWWS���ELW�O\�VV�����RR\DOD