AWS Black Belt Online Seminar 2017 Amazon EMR

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

w

, +1(++( +

r4JF 5YNP 5RYa B YV R FRZV N_s4ZNg 8@E


•z

•w× w

w w ×

• 7MI w7 Rk _ CH& 7 Rk _ 9I


v

• – ()/ )) ) n x v v k–4JF Uaa 2 Nd &NZNg &P Z# vk

• – x zj 4JFzu j4JF v k

• – v k x z n jv k

• 7MI U Vd _ e WWVc SZ_UZ_X acZTV bf eVd( 7MI acZTZ_X Zd afS ZT j RgRZ RS V R_U ZddfS VTe e TYR_XV Z_ RTT cUR_TV hZeY eYV 7MI 9fde Vc 7XcVV V_e RgRZ RS V ReYeea4))Rhd(R Rk _(T )RXcVV V_e)( 7_j acZTZ_X Z_W c ReZ _ Z_T fUVU Z_ eYZdU Tf V_e Zd ac gZUVU _ j Rd R_ VdeZ ReV W fdRXV TYRcXVd W c 7MI dVcgZTVd SRdVU_ TVceRZ_ Z_W c ReZ _ eYRe j f YRgV ac gZUVU( C _eY j TYRcXVd hZ SV SRdVU _j fc RTefR fdV W 7MI dVcgZTVd& R_U Rj gRcj Wc eYV VdeZ ReVd ac gZUVU(


4TR QN

•• 7 Rk _ CH• 7 Rk _ CH w , +0)+ $• 7 Rk _ CH .(i• w 7 Rk _ CH•


–~


4ZNg 8@E❏ , 3 w

• CRaHVUfTV p +

❏ >RU a ×

• FZX& >ZgV& >8RdV& FcVde & >fV&>RU a ,(i& IaRc ,(i& Z_

❏ 7MI w• Ia e& 7C& :j_R :8& LF9&

9 fUMReTY& 9B & AZ_VdZd& ACI

→ – @N ERQbPR– vm

Storage S3 (EMRFS), HDFS

YARNCluster Resource Management

BatchMapReduce

InteractiveTez

In MemorySpark, Flink

ApplicationsHive, Pig, Spark SQL/Streaming/ML, Mahout, Sqoop

HB

ase/ Phoenix

Presto

Hue (SQL Interface/Metastore Management)Zeppelin (Interactive Notebook)

Ganglia (Monitoring)HiveServer2/Spark Thriftserver (JDBC/ODBC)

Amazon EMR service


n

S3

EMR

EMR Redshift

MachineLearning

Collect Process AnalyzeStore

Data Collectionand Storage

DataProcessing

EventProcessing

Data Analysis

Data Answers

Athena

Quicksight

Kinesis

DynamoDB

AWSIoT

AWSLambda

KCLApps Glue


❏ w)

• CRdeVc)9 cV)JRd _ UV• CH I• 8 edecRa 7TeZ _& IeVad

n❏

w• >RU a& O7HD• >ZgV& FZX& >8RdV& IaRc &

Z_ & FcVde & VeT(

4ZNg 8@E


4ZNg 8@En


Task Node

Task Instance Group

4ZNg 8@E n

security group

security group

Master Node

Master Instance Group

Amazon S3

AmazonDynamoDB

Amazon Kinesis

Core Node

Core Instance Group

HDFS HDFS

HDFS HDFS

Task Node

Task Instance Group

w >: I

7MI w

AWS GlueData Catalog


4ZNg 8@E @N aR_ aN PR :_ b

❏CRdeVc D UV +• RZ gVc

❏ w

• DR VD UVHVd fcTVCR_RXVc

• 9 cV D UV JRd D UV

Master Node

Master Instance Group

Hadoop1: JobTrackerHadoop2: ResourceManagerHDFS: NameNodeHive: HiveServer2, MetaStorePresto: Coordinator


4ZNg 8@E 6 _R aN PR :_ b

❏ + 9 cV D UV❏ w

• D UVCR_RXVc• :ReRD UV w

>: I

• 8IpEA

HDFS

Core Node

Core Instance Group

Hadoop1: TaskTrackerHadoop2: NodeManagerHDFS: DataNodePresto: Worker


4ZNg 8@E 6 _R aN PR :_ b

❏ 9 cV D UV• >: I• 9FK)H7C

❏>: Ip

• .(+(

HDFS

Core Node

Core Instance Group

Hadoop1: TaskTrackerHadoop2: NodeManagerHDFS: DataNodePresto: Worker


4ZNg 8@E GN aN PR :_ b

❏ >: I9 cV

❏ >: I w 9 cVD UV

❏ >: I

• >: I w:VT ZddZ _$

Task Node

Task Instance Group

Hadoop1: TaskTrackerHadoop2: NodeManagerPresto: Worker


4ZNg 8@E GN aN PR :_ b

❏ =c fa• Ia e SZU• _deR_TV JjaV

❏ HpIa e

Task Instance Group 2

Task Instance Group 1

c3.xlarge * 2bid: $0.1

r3.xlarge * 2bid: $0.5


F a aN PR

Task Instance GroupCore Instance Group

❏ ❏

❖ B QRZN Q9 cV D UV

❖ ”×

E_ UV R_U

❖ F a aN PR JRdD UV

❖ E_ UV R_U3

w

4 .(i RcXV 0 + +, p/ $6 _R A QR B QRZN Q# 4 (-+2o+ Yo0GN A QR F a# 4 (-+2o/Yo0 % ( .o/Yo0

, +1 ++ + $ w qIa e q

5 +3( 2 +5 + (1. /


4ZNg 8@El


8@E9F2 4ZNg F 79F w

❏ w❏

• w

❏ w• w

❏ w

EMR

EMR

Amazon S3


8@E9F

❏ ld-4))n ”❏7 Rk _ I-

• 4 w 7 Rk _ = RTZVc

❏7 Rk _ I- w s

• 7 Rk _ I- w

❏ w


8@E9F 6 V aR a IVRd❏ 7 Rk _ I-

• FKJHVRU RWeVc hcZeV T _dZdeV_Tj$

• FKJ): B J gV_efRT _dZdeV_Tj$

❏ CH I 9 _dZdeV_e LZVh• 7 Rk _ :j_R :8 w

• BZde 9 _dZdeV_Tj• :V VeV 9 _dZdeV_Tj• HVRU RWeVc hcZeV 9 _dZdeV_Tj

•

Amazon S3 Amazon DynamoDB

CH Iw

Yeead4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)CR_RXV V_e=fZUV)V c a R_ T _dZdeV_e gZVh(Ye


4ZNg 8@E – v

❏ ”••

❏ w 7 Rk _ I-• w

• w

t


4ZNg 8@E 2 5 a a_N 4PaV

❏ D UV• EA

• 8RdY& HfSj& FjeY _& VeT(• 7 Rk _ I- p

• 7aRTYV 8ZXe a ww p8 edecRa

Yeea4))Rhd(ejaVaRU(T )dR a), +0) 3)SfZ UZ_X R_U UVa jZ_X Tfde Raa ZTReZ _d hZeY RaRTYV 8ZXe a R_U R Rk _ V c(Ye


4ZNg 8@E 2 FaR

❏ )• p

• 4 JB >ZgVGB

❏ 7 Rk _ I- KH p p w• I- Rc• IecVR Z_X& >ZgV& FZX& IaRc CH w T R_U cf__Vc( Rc $• dTcZae cf__Vc( Rc SRdY

❏ IeVa7fe eVc Z_ReV$


4ZNg 8@Ey

❏ IeVa• 7 Rk _ CH

• w w 9B• 7MI I:A• 7C

• )•

❏ 7MI w• BR SUR• :ReR FZaV Z_V

❏ w

Amazon EMR

Amazon EMR Step API

AWS Data Pipeline

Airflow, Luigi, or other schedulers on EC2

AWS Lambda


4ZNg 8@Ey

❏ w w• CRdeVc D UV ddY p pIIC 7 Rk _ 9, IjdeV d CR_RXVc$ Hf_ T R_U

• >ZgVIVcgVc,& daRc dfS Ze• PVaaV Z_& >fV& daRc SdVcgVc =K• BZgj HVde IaRc• E kZV w w

Web UIs: Hue SQL editor, Zeppelin notebooks,

R Studio, and more!

Connect with ODBC / JDBC using HiveServer2/Spark Thriftserver

Use Spark Actions in your Apache Oozie workflow to create DAGs of jobs.

(start using start-thriftserver.sh)

Or, use the native APIs and CLIs for each application


C_VcNaR FbO Ra 8@E z

❏ IfS_Ve FcZgReV

• CH .(,(

❏ FfS ZT• I-):j_R :8LF9 V_Ua Z_e

• I-):j_R :8 wD7J =M

• CH IVcgZTVIVTfcZej =c fa

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +/)+,) Rf_TY R Rk _ V c T fdeVcd Z_ R Rk _ gaT acZgReV dfS_Ved)


4 NPUR F N_ GRg NQ @N ERQbPRC_R a n z

Yeea4))U Td(Rhd(R Rk _(T )V c) ReVde)HV VRdV=fZUV)V c UReR V_TcjaeZ _ aeZ _d(Ye

❏ Re cVde pZ_ ecR_dZe pw

❏ Re cVde• CH I& >: Ip p w

× w❏ Z_ ecR_dZe

• JBI❏ CH ×

• 9 fUW c ReZ _


4ZNg 8@E n (). )( #


4ZNg 8@E n (). )( #

❏ 7aRTYV Z_ w❏ w❏ w w w❏ 7aRTYV >8RdV w I-❏ 7 Rk _ 9 fUMReTY gV_ed❏ w w❏ 7C w w❏ Z- dVcZVd 7 Rk _ 9, Z_deR_TVd w❏ 7 Rk _ CH w


4 NPUR 9YV n

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)_ h cf_ cVR eZ V decVR ac TVddZ_X Re dTR V hZeY RaRTYV W Z__ R Rk _ V c)

❏ HV VRdV /(+( w❏ HV VRdV /(3( p7aRTYV Z_ +(-(,

• w p

• IaRc p pIGB

• 7 Rk _ AZ_VdZd IecVR d w• w• ViRTe j _TV ×• wp × w


ny z

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)R Rk _ V c _ h dfaa ced WZ_V XcRZ_VU RTTVdd T _ecfdZ_X T fdeVc eRXd)

❏ CH w p 7Cw

– + +( :Vae5: L w p×

– + ,( :Vae5: Lp pJVdew pz

– , +( :Vae5: L p w

– , ,( :VaezLR fV pz

❏ w w 9 fdeVc :

– ( :VdTcZSV9 fdeVc pBZde9 fdeVcd


n n n

❏ HV VRdV .(i /(i w❏ w

– HV VRdV /(+(• p

pw

p w•

– HV VRdV /(+(• p w

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)R Rk _ V c _ h dfaa ced Rfe dTR Z_X R_U T _WZXfcRS V dTR VU h_ SVYRgZ c)


4 NPUR 5N R nF z

❏ HV VRdV /(,( w– w p>8RdV w

I- >: I– w I- w

❏ HV VRdV /(1(– >8RdV I-

w

❏ KF– CH w w I-– CH I 9 _dZdeV_e LZVh– M7B Z_ V cj W c McZeV– /=8 c Ie cV– Z_ V cj) _ UZd W c HVRU

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)R Rk _ V c _ h dfaa ced fdZ_X R Rk _ d- Rd R UReR de cVW c RaRTYV YSRdV)


4ZNg 6Y bQJNaPU 8cR a

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)+,)R Rk _ V c _ h dfaa ced VgV_ed R_U _ eZWZTReZ _d hZeY R Rk _T fUhReTY VgV_ed)

❏ CH– w

• IJ7HJ D=& HKDD D=& M7 J D=& J HC D7J :& J HC D7J :QM J>Q HHEHI

– w• Wc H I P D= e HKDD D=z & Wc HKDD D= e H I P D=z

– w w w• F D: D=& 7JJ79> :& : J79> :& 7 B :

– ×• F D: D=& 97D9 BQF D: D=& HKDD D=& 9ECFB J :& 97D9 BB :& 7 B :


n n

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +1) -)R Rk _ V c R__ f_TVd Z_deR_TV W VVed W c R Rk _ VT, da eZ_deR_TVd R_U da e S T d)

❏ p CH w❏

– w p / r× r

–– p+ 0– C9 × g9FK

• g9FK p 9, g9FK pCH O7HD D UVCR_RXVc gT cV

– w w• w × p w w

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +1) -)R Rk _ V c R__ f_TVd Z_deR_TV W VVed W c R Rk _ VT, da eZ_deR_TVd R_U da e S T d)

n n

+( ×,(-(.( c ×/( 8I w0(1( g9FK

�

�

�

�

�

�

��


4ZNg V be 4@n n

Yeead4))Rhd(R Rk _(T ) a)RS fe Rhd)hYRed _Vh), +1) 1)R Rk _ V c _ h dfaa ced Rf_TYZ_X T fdeVcd hZeY TfdeR Rk _ Z_fi R Zd)

❏ 7 Rk _ BZ_fi 7 Rk _ CRTYZ_VRXV 7C $ 7 Rk _ CH w

❏ 7C w pw 7C

❏ w /(1( 7 Rk _ CH w


V n 4ZNg 86 V aN PR nBPa OR_ ) ()/#

Yeead4))Rhd(R Rk _(T ) a)RS fe Rhd)hYRed _Vh), +1)+ )R Rk _ V c _ h dfaa ced Z- Z_deR_TVd)

❏ w 7 Rk _ 9, wZ-

❏ Z- pD _ L ReZ V CV cj iacVddDLCV$ II: w p E

E❏ w /(3( 7 Rk _ CH w


4ZNg 8@E n

Yeead4))Rhd(R Rk _(T ) a)RS fe Rhd)hYRed _Vh), +1)+ )R Rk _ V c _ h dfaa ced aVc dVT _U SZ Z_X)

❏ , +1 + , p 7MI w

❏ ww

❏ w +

❏ p++ $


4ZNg 8@E ,&e


4ZNg 8@E ERYRN R ()- / #• HV VRdV .( ( m , +/) 1• HV VRdV .(+( m , +/) 3• HV VRdV .(,( m , +/)++• HV VRdV .(-( m , +0) +• HV VRdV .(.( m , +0) -• HV VRdV .(/( m , +0) .• HV VRdV .(0( m , +0) .• HV VRdV .(1(i m , +0) 0 1• HV VRdV .(2(i m , +0) 2 , +1) ,• HV VRdV .(3(+ m , +1) -• HV VRdV .(3(, m , +1) 1

7aRTYV 8ZXe a w w ×+, +, w

>RU a ×Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV)V c cV VRdV T a _V_ed(Ye

• HV VRdV /( ( m , +0) 2• HV VRdV /( (- m , +0)+• HV VRdV /(+( m , +0)++• HV VRdV /(,(i m , +0)+,• HV VRdV /(-(i m , +1) + ,• HV VRdV /(.( m , +1) ,• HV VRdV /(/( m , +1) .• HV VRdV /(0( m , +1) 0• HV VRdV /(1( m , +1) 1• HV VRdV /(2( m , +1) 2• HV VRdV /(3( m , +1)+


4ZNg 8@E ERYRN R

❏ w• 7 Rk _ CH p w w

❏ w• p w

❏ w w p• GfZT 9cVReV• pCR_RXV V_e =fZUV HV VRdV =fZUV

7C w w phttp://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-release-differences.html


ERYRN R2 n

❏ w w• w• 4

• jRc_(cVd fcTV R_RXVc(hVSRaa(RUUcVdd a ce 3 ,0 56 2 22• )Y V)YRU a)YZgV 56 )fdc) ZS)YZgV& )VeT)YZgV)T _W& )gRc) X)YZgV

❏ w w w• YRU a w ”• jRc_ w pYZgV w


ERYRN R2 n

❏ 9 RddZWZTReZ _ % Fc aVceZVd• 8 edecRa 7TeZ _• w ”

Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV)V c T _WZXfcV Raad(Ye

e.g. /etc/hadoop/conf/mapred-site.xml


ERYRN R2 n j

❏ w w GfZT 9cVReV• p w p × - ”• w = e RUgR_TVU aeZ _d

❏ w• CR_RXV V_e =fZUV

• 7 Rk _ CH w w• Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)CR_RXV V_e=fZUV

• HV VRdV =fZUV• w w• Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV


n o-&1&(#

Yeea4))U Td(Rhd(R Rk _(T ))V c) ReVde)HV VRdV=fZUV)V c cV VRdV T a _V_ed(Ye


ERYRN R ,&ey -&e

❏ w• IaRc ,(i

• FPNYN &))• >ZgV ,(i• Z_• BZgj

❏

• NcN 0• JVk >ZgV& FZX$

❏

• IeVa

❏ IR_US i• w

• IR_US i


4ZNg 8@E 4 YVPNaV S _ -&1&(

• >RU a& >ZgV& JVk&IaRc & FcVde & FZX&CRY fe& Ib a& Z_

w w• >8RdV& FY V_Zi

=K• >fV& PVaaV Z_

w w• E kZV

• =R_X ZR& >9ReR X&P VVaVc& YVcf


4 YVPNaV 2

❏ >RU a• qO7HD w $%>: I w $ p

qCRaHVUfTV + q CHO7HD p q

❏ >ZgV• O7HD p IGB q

CRaHVUfTV qI- w q

❏ JVk• O7HD :7= $ q

>ZgV ” p >ZgV q


4 YVPNaV 2

❏ IaRc• w w qO7HD pCRaHVUfTV

qIaRc IaRc IGB >ZgV $ IaRcIecVR Z_X w $& CB ZS $ p

q❏ Z_

• w w q O7HD p wp qIaRc p

pIGB qw ViRTe j _TV × q

❏ FcVde• IGB qO7HD ” pIGB

q w E D qIGB7DI IGB q


4 YVPNaV 2

❏ FZX• CRaHVUfTV :IB q

q

❏ CRY fe• CRaHVUfTV IaRc q

q

❏ Ib a• H:8CI p>: I I- w CRaHVUfTV

q


4 YVPNaV 2 n n

❏ >8RdV• >: I ALIq w

w q

❏ FY V_Zi• >8RdV w IGB w q


4 YVPNaV 2 :H n n

❏ >fV• >ZgV ×w p =K q

❏ PVaaV Z_• =K IaRc w IaRc IGB p

p s w q

❏ E kZV• w w w qCRaHVUfTV>ZgV q


4 YVPNaV 2

❏ =R_X ZR• w 9FK&CV cj&DVeh c $

p q

❏ >9ReR X• >ZgV ×w FZX w q

❏ P VVaVc• w w pALI p w w

q >8RdV $ q


4 YVPNaV 2

❏ BZgj• IaRc CH w H IJ w p

pHF9w w


n 4ZNg 8@E


4ZNg 8@E x n

❏ w w•

” w• HV VRdV BRSV )7C LVcdZ _

w p w

❏ w w• 7aRTYV 8ZXe a

w wp8 edecRa

• w7C

Yeea4))Rhd(ejaVaRU(T )dR a), +0) 3)SfZ UZ_X R_U UVa jZ_X Tfde Raa ZTReZ _d hZeY RaRTYV 8ZXe a R_U R Rk _ V c(Ye


n 2 NQ

❏ O7HD 5 OVe 7_ eYVc HVd fcTVDVX eZRe c

❏ >RU a, w

• SJcRT Vc

❏ HVd fcTV HVbfVde w w• 7aa ZTReZ _CRdeVc

p wHVd fcTVCR_RXVc

MapReducePig Hive Other

Other

HDFS

OtherMapReducePig Hive

YARNHDFS

Hadoop V1

Hadoop V2


K4EA2 ER b_PR@N NTR_

❏ w w❏ w w

• 9FK& CV cj& VeT(❏ w

❏

• 7aa ZTReZ _CRdeVc

Yeea4))YRU a(RaRTYV( cX)U Td)TfccV_e)YRU a jRc_)YRU a jRc_ dZeV)O7HD(Ye


K4EA2 A QR@N NTR_

❏ w w❏ w w

HC❏ w 9 _eRZ_Vc

• 9 _eRZ_Vc



K4EA2 6 aNV R_

❏ w w

❏ DC❏

• :VWRf e9 _eRZ_Vc iVTfe c•

• BZ_fi9 _eRZ_Vc iVTfe c• TXc fad

• : T Vc9 _eRZ_Vc iVTfe c• : T Vc



K4EA2 4 YVPNaV @N aR_

❏9 _eRZ_Vc• SJcRT Vc• + 7C

❏9 _eRZ_Vc



4ZNg 8@E K4EA

❏CRaHVUfTV& >ZgV& FZX& IaRc _ O7HD• 7 Rk _ CH pO7HD w

❏ ×• CH >ZgV IaRc p w


n 2 VcR

❏ IGB Z V p ww

❏ w w w p×w• >: I w w• 7 Rk _ CH CH I 7 Rk _ I-$

• p7 Rk _ AZ_VdZd& 7 Rk _ :j_R :8

❏• CRaHVUfTV& JVk& IaRc

Yeead4))YZgV(RaRTYV( cX)


VcR

Metastore

HDFS

Cluster

Amazon S3

HiveServer2CREATE TABLE

SELECTFROM

+4 ×w>: I 7 Rk _ I-

w

,4 ×ww

-4 ×w.4 ×w

w

/4

04 ww

AmazonDynamoDB

Amazon Kinesis


VcR @RaN a _R

❏ IGBCVeRde cV

❏ CVeRde cV CjIGB

• 7 Rk _ CH CRdeVcD UV w

• p7 Rk _ H:I❏ w 7MI = fV

:ReR 9ReR X CVeRde cV

CVeRde cV

Amazon S3

HDFS


v n BE6 9VYR CN_ bRa

❏ w• w•

• EaeZ ZkVU H h 9 f _Rc EH9$ Z V ) FRcbfVe

❏ w• +• + p w

• J NJ B & I GK D9 B EH9 wYeead4)) cT(RaRTYV( cX)U Td)daVT Z_ec (Ye


n w

❏

• w• w w

❏ w

❏ w p+ w +


VcR BE6 9VYR CN_ bRa v

❏×w”

❏

CREATE TABLE t (col1 STRING,…

) STORED AS [ORC/PARQUET];

INSERT INTO t (…);

SELECT col1 FROM t;


VcR 4ZNg V R V n

❏ 7 Rk _ AZ_VdZd IecVR w p>ZgV ×w

• 7 Rk _ AZ_VdZd p w w• 4 >ZgV p w

❏ IecVR IYRcU CRaaVc w• 7 Rk _ AZ_VdZd w• 7 Rk _ :j_R :8 $ p ”

w

IYRcU

IYRcU +

CRaaVc

CRaaVc

SELECTFROM

123

id data

1 …

2 …

3 …Amazon Kinesis


VcR 4ZNg V R V nVcR n

7 Rk _ AZ_VdZdIecVR

D #

>GB ” q7 Rk _ AZ_VdZd

w


n 2 F N_

❏w w

• p

• $

❏ w w pw

• IGB& w & &

http://spark.apache.org/


F N_ FD 7NaN Ra 4C F N_ & #

❏ IaRc IGB• >ZgVGB p>ZgV CVeRde cV• :89)E:89• IGB p

:ReRdVe):ReR cR V

❏ :ReRdVe 7F• w• ITR R) RgR• :ReR cR V

:ReRdVeYeead4))UReRScZT d(T )S X), +0) 1)+.)R eR VW eYcVV RaRTYV daRc RaZd cUUd UReRWcR Vd R_UUReRdVed(Ye

Yeea4))daRc (RaRTYV( cX)U Td) ReVde)db ac XcR Z_X XfZUV(Ye


❏ :ZdTcVeZkVU IecVR ��$ p❏ HVdZ ZV_e :ZdecZSfeVU :ReRdVe ��$ w

• p

❏ pIaRc ,(, CH HV VRdV /(2( $ pIecfTefcVUIecVR Z_X 7F =7

DStreamRDD@T1 RDD@T2

MessagesReceiver

F N_ Fa_RNZV T i

Yeea4))daRc (RaRTYV( cX)U Td) ReVde)decVR Z_X ac XcR Z_X XfZUV(YeYeead4))daRc (RaRTYV( cX)U Td) ReVde)decfTefcVU decVR Z_X ac XcR Z_X XfZUV(Ye


4ZNg V R V F N_ Fa_RNZV T

7 Rk _ AZ_VdZd IecVR p7 Rk _ CH IaRc:IecVR

• IaRc 7 Rk _ AZ_VdZd• pAZ_VdZd 9 ZV_e BZScRcjzA9B

x p• w p

http://spark.apache.org/docs/latest/streaming-kinesis-integration.html

IYRcU

IYRcU +

A9B eYcVRU

A9B eYcVRUAmazon Kinesis

DStreamRDD@T1 RDD@T2


F N_ 7f NZVP ER b_PR 4YY PNaV

❏ iVTfe c H::• >VRa p

❏ :j_R ZT HVd fcTV 7 TReZ _ 5 iVTfe c

• IecVR Z_X 7aa ZTReZ _ w• daRc dYV w• .(.( pdaRc (Uj_R ZT7 TReZ _(V_RS VU

a_bR• daRc (dYfWW V(dVcgZTV(V_RS VU a_bR

https://spark.apache.org/docs/latest/job-scheduling.html


❏ w)

n❏

w

4ZNg 8@E

w p + “ u


p q• 7 Rk _ CH CR_RXV V_e =fZUV

Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)CR_RXV V_e=fZUV

• 7 Rk _ CH HV VRdV =fZUVYeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV

• 7 Rk _ CH 7F HVWVcV_TVYeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)7F )

• 7 Rk _ CH :VgV aVc =fZUV ,(i R_U -(i $Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde):VgV aVc=fZUV)

• 7MI 9B V cYeea4))U Td(Rhd(R Rk _(T )T Z) ReVde)cVWVcV_TV)V c)Z_UVi(Ye


D 4


n7MI w

• Yeea4))Rhd(R Rk _(T ) a)Rhd a Z_ec UfTeZ _)

7MI I feZ _d 7cTYZeVTe• p w G 7• Yeea4))Rhd(ejaVaRU(T )dR a)


GdVaaR_ 9NPRO4JF x

3Nd PY bQMW

p p pw v u

–Uaa 2 &SO&ZR )cE0fJZ


4JF jx v7MI p p

”Yeead4))Rhd(R Rk _(T ) a)T _eRTe fd)Rhd dR Vd)

7MI ”


4 R QVe


4@ E YR z

, +/ 0 - p 7C H V

7C H V , p• CH c V

• 7 Rk _ CH p7 Rk _ 9,• 9, Z_deR_TV ac WZ V

• 7 Rk _ 9, p7 Rk _ I-

Yeea4))Rhd(ejaVaRU(T )dR a), +/) 0)V c R_URe cj ZR faUReV(Ye


ERYRN R ,&(

w w• >RU a ,(0(• >ZgV +( (• FZX (+.(• CRY fe (+ (• IaRc +(.(+

7aRTYV 8ZXe a w w ×• p


ERYRN R ,&)

w• IaRc +(/(• >fV -(1(+ $

p w• FcVde (++3• PVaaV Z_ (0 I_RadY e$• E kZV .( (+


ERYRN R ,&

w• IaRc +(/(,• =R_X ZR -(0( $

• FcVde (+,/• PVaaV Z_ (/(/• E kZV .(,(


ERYRN R ,&

w• >RU a ,(1(+• IaRc +(0(• =R_X ZR -(1(,

• FcVde (+-


ERYRN R ,&,

w• >9ReR X +( ( $• CRY fe (++(+

• Ib a +(.(0 $• FcVde (+-0• PVaaV Z_ (/(0

IaRc• :j_R ZT R TReZ _

RgR 2• 7L7Q>EC ”


ERYRN R ,&-

w• IaRc +(0(+• >RU a ,(1(,

• FcVde (+.

CH I• I- II ACI

• 7MI ACI 9CA


ERYRN R ,&.

w• >8RdV +(,( $ m D IGB _ >RU a

• P VVaVc -(.(2 $• FcVde (+.-


ERYRN R ,&/&) t zu ,&/&(–

w• FY V_Zi .(1( $ m IGB _ >8RdV• JVk (2(- $ m :7= V_XZ_V _ O7HD• >8RdV +(,(+• CRYRfe (+,(

• FcVde (+.1


ERYRN R ,&/&

w• CRY fe (+,(,• IaRc +(0(,

• FcVde (+.2


ERYRN R ,&0&(

w• >8RdV +(,(,• JVk (2(.

• PVaaV Z_ (0(+• FcVde (+/+


ERYRN R ,&0&

w• >RU a ,(1(-

• FcVde (+/,(-


ERYRN R ,&0&

w• IaRc +(0(-

• FcVde (+/1(+• P VVaVc -(.(3

• CH :j_R :89 __VTe c– .(,(– >ZgV& IaRc


ERYRN R ,&1&)

w•

FjeY _ ,(0 w• w

>ZgV• > L 3310• > L + + 0


ERYRN R -&(&(

w• >ZgV ,(+(• FcVde (+/• IaRc ,( (

• FPNYN &))• >fV -(+ (• FZX (+0(• JVk (2(.• >8RdV +(,(,• PVaaV Z_ (0(+

• NcN 0• JVk >ZgV& FZX$

• IeVa

IR_US i• w

• IR_US i


85F zj @, 6, n

CH 7F 8I

• >: I

• p

• 9 fU c ReZ _

8I I9+)IJ+ w• >: I

C. 9.

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0) ,)R Rk _ V c _ h dfaa ced R Rk _ VSd g f Vd R_U . R_UT. R Rk _ VT, WR Z ZVd)


4 NPUR VcR 4 NPUR F N_ 8@E7f NZ 75 n BFF:j_R :8:j_R :8 ×w p CH w

E D=Ze>fS

• Yeead4))XZeYfS(T )Rhd RSd)V c Uj_R US T __VTe c

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0) 3)eYV R Rk _ V c Uj_R US T __VTe c W c RaRTYV YZgV R_URaRTYV daRc Zd _ h aV_ d fcTV)


ERYRN R -&(&

w• >RU a ,(1(-• FcVde (+/,(-• IaRc ,( (+


ERYRN R -&)&(

w• Z_ +(+(- $• >8RdV +(,(-• PVaaV Z_ (0(,


ERYRN R -& &(

w• IaRc ,( (,

>8RdV _ I-• >8RdV w

I-


ERYRN R -& &)

w• FcVde (+/1(+• P VVaVc -(.(3

• CH :j_R :89 __VTe c– .(,(– >ZgV& IaRc


ERYRN R -& &(

w• >ZgV ,(+(+• >fV -(++(• IaRc ,(+(• E kZV .(-(• Z_ +(+(.


ERYRN R -& &)

w•

PVaaV Z_• P FF B D +2/0• P FF B D +311


ERYRN R -&,&(

w• FcVde (+00• FY V_Zi .(3(• >8RdV +(-(• Z_ +(,(• PVaaV Z_ (1(


E, n

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +1) -)R Rk _ V c cV VRdV / . R_U dfaa ce W c c. Z_deR_TVd _ hRgRZ RS V)

H.• × w w


ERYRN R -&-&(

w• >fV -(+,• FcVde (+1

• B:7F w• PVaaV Z_ (1(+• P AVVaVc -(.(+


ERYRN R -&.&(

w• >8RdV +(-(+• IaRc ,(+(+• Z_ +(,(+• CRY fe (+-(

• IaRc ,(i w

FcVde Z_ ecR_dZe


ERYRN R -&/&(

w• Z_ +(-(• FY V_Zi .(++(• PVaaV Z_ (1(,

w p7 Rk _ BZ_fi 7C

5N R• w w

C_R a• _ UV(ac aVceZVd

K4EA• T _eRZ_Vc X. (ac aVceZVd

F• IGEEF ,22 w


ERYRN R -&0&(

w• 7MI I:A +(++(+0• Z_ +(-(+• >ZgV ,(-(• IaRc ,(,(

• IecfTefcVU IecVR Z_X =7

7MI = fV :ReR 9ReR X

• >ZgV IaRc IGB 7MI= fV :ReR 9ReR X >ZgV

w

B gVR• EEP ,1.2 wbR• >K /2/3 w5N R• CN $ >8RdV w

w w

• w

RgR CR_RXV V_e ieV_dZ _d


ERYRN R -&1&(

w• 7MI I:A W c RgR +(++(+2-• Z_ +(-(,• >fV .( (+

• >fV D eVS W c IaRc w• FZX (+1(• FcVde (+2.• BZgj (.( Z_TfSReZ_X $

Z- dVcZVd 7 Rk _ 9, Z_deR_TVdw ETe SVc +,& , +1$


ERYRN R -&1&(

F N_• w p

p7fe ITR Z_XXcRTVWf

• IF7HA ,+.3. wLR RYV

• P FF B D ,-11 wON R

• >87I +2/--

bR• >ZgV bfVcj VUZe c7MI = fV :ReR 9ReR X

w•

AWS Black Belt Online Seminar 2017 Amazon EMR

Technology

Transcript of AWS Black Belt Online Seminar 2017 Amazon EMR