final sludge - Cornell Waste Management Institute - Cornell University
MapReduce - Cornell University
Transcript of MapReduce - Cornell University
MapReduce SimplifiedDataProcessingonLargeClusters
(WithouttheAgonizingPain)
PresentedbyAaronNathan
TheProblem
• Massiveamountsofdata– >100TB(theinternet)– Needssimpleprocessing
• Computersaren’tperfect– Slow– Unreliable– Misconfigured
• Requirescomplex(i.e.bugprone)code
MapReducetotheRescue!
• CommonFuncKonalProgrammingModel– MapStep
map (in_key, in_value) -> list(out_key, intermediate_value) • Splitaproblemintoalotofsmallersubproblems
– ReduceStepreduce (out_key, list(intermediate_value)) -> list(out_value) • Combinetheoutputsofthesubproblemstogivetheoriginalproblem’sanswer
• Each“funcKon”isindependent• HighlyParallelizable
Answer!Answer!
AlgorithmPicture
MAP
REDUCE
Answer!
MAPMAP MAP MAP
DATA
K1:vK1:vK2:v K2:v K1:v K2:vK3:v K3:v
K1:v,v,v K2:v,v,v K3:v,v,v
REDUCE REDUCE
aggregator
SomeExampleCodemap(String input_key, String input_value): // input_key: document name
// input_value: document contents
for each word w in input_value: EmitIntermediate(w, "1");
reduce(String output_key, Iterator intermediate_values): // output_key: a word
// output_values: a list of counts int result = 0; for each v in intermediate_values:
result += ParseInt(v);
Emit(AsString(result));
SomeExampleApplicaKons
• DistributedGrep• URLAccessFrequencyCounter• ReverseWebLinkGraph
• Term‐VectorperHost
• DistributedSort• InvertedIndex
TheImplementaKon
• GoogleClusters– 100s‐1000sDualCorex86CommodityMachines
– CommodityNetworking(100mbps/1Gbps)– GFS
• GoogleJobScheduler• Librarylinkedinc++
ExecuKon
TheMaster
• MaintainsthestateandidenKfyofallworkers• Managesintermediatevalues
• ReceivessignalsfromMapworkersuponcompleKon
• BroadcastssignalstoReduceworkersastheywork
• CanretaskcompletedMapworkerstoReduceworkers.
InCaseofFailure
• PeriodicPingsfromMaster‐>Workers– Onfailureresetsstateofassignedtaskofdeadworker
• Simplesystemprovesresilient– Worksincaseofa80simultaneousmachinefailures!
• Masterfailureisunhandled.• WorkerFailuredoesn’teffectoutput
(outputidenKcalwhetherfailureoccursornot)– Eachmapwritestolocaldiskonly– Ifamapperislost,thedataisjustreprocessed– Non‐determinisKcmapfuncKonsaren’tguaranteed
PreservingBandwidth
• Machinesareinrackswithsmallinterconnects– UselocaKoninformaKonfromGFS
– Ahemptstoputtasksforworkersandinputslicesonthesamerack
– UsuallyresultsinLOCALreads!
BackupExecuKonTasks
• Whatifonemachineisslow?• CandelaythecompleKonoftheenKreMROperaKon!
• Answer:Backup(Redundant)ExecuKons– Whoeverfinishesfirstcompletesthetask!
– Enabledtowardstheendofprocessing
ParKKoning
• M=numberofMapTasks(thenumberofinputsplits)
• R=numberofReduceTasks(thenumberofintermediatekeysplits)
• W=numberofworkercomputers• InGeneral:
– M=sizeof(Input)/64MB– R=W*n(wherenisasmallnumber)
• TypicalScenario:InputSize=12TB,M=200,000,R=5000W=2000
CustomParKKoning
• DefaultParKKonedonintermediatekey– Hash(intermediate_key)modR
• Whatifuserhasaprioriknowledgeaboutthekey?– Allowforuser‐definedhashingfuncKon– Ex.Hash(Hostname(url_key))
TheCombiner
• IfreducerisassociaKveandcommuniviKve– (2+5)+4=11or2+(5+4)=11– (15+x)+2=2+(15+x)
• Repeatedintermediatekeyscanbemerged– Savesnetworkbandwidth– EssenKallylikealocalreducetask
I/OAbstracKons
• HowtogetiniKalkeyvaluepairstomap?– Defineaninput“format”
– Makesuresplitsoccurinreasonableplaces– Ex:Text
• Eachlineisakey/pair• CancomefromGFS,bigTable,oranywherereally!
– Outputworksanalogously
SkippingBadRecords
• Whatifausermakesamistakeinmap/reduce• Andonlyapparentonfewjobs..
• WorkersendsmessagetoMaster
• Skiprecordon>1workerfailureandtellotherstoignorethisrecord
RemovingUnnecessaryDevelopmentPain
• LocalMapReduceImplementaKonthatrunsondevelopmentmachine
• MasterhasHTTPpagewithstatusofenKreoperaKon– Shows“badrecords”
• ProvideaCounterFacility– Masteraggregates“counts”anddisplayedonMasterHTTPpage
AlookattheUI(in1994)
h6p://labs.google.com/papers/mapreduce‐osdi04‐slides/index‐auto‐0013.html
PerformanceBenchmarks
SorBng AND Searching
Search(Grep)
• Scanthrough1010100byterecords(1TB)
• M=15000,R=1
• StartupKme– GFSLocalizaKon– ProgramPropagaKon
• Peak‐>30GB/sec!
Sort
• 50linesofcode• Map‐>key+textline
• Reduce‐>IdenKty• M=15000,R=4000
– ParKKononinitbytesofintermediatekey
• Sortsin891sec!
WhataboutBackupTasks?
Andwait…it’suseful!
NB:August2004
OpenSourceImplementaKon
• Hadoop• hhp://hadoop.apache.org/core/
– ReliesonHDFS– AllinterfaceslookalmostexactlylikeMapReducepaper
• Thereisevenatalkaboutittoday!– 4:15B17CSColloquium:MikeCafarella(Uwash)
AcBveDisksforLarge‐ScaleDataProcessing
TheConcept
• Useaggregateprocessingpower– Networkeddisksallowforhigherthroughput
• WhynotmovepartoftheapplicaKonontothediskdevice?– Reducedatatraffic
– Increaseparallelismfurther
ShrinkingSupportHardware…
ExampleApplicaKons
• MediaDatabase– Findsimilarmediadataby“fingerprint”
• RealTimeApplicaKons– CollectmulKplesensordataquickly
• DataMining– POSAnalysisrequiredadhocdatabasequeries
Approach
• Leveragetheparallelismavailableinsystemswithmanydisks
• Operatewithasmallamountofstate,processingdataasitstreamsoffthedisk
• ExecuterelaKvelyfewinstrucKonsperbyteofdata
Results‐NearestNeighborSearch
• Problem:DeterminekitemsclosesttoaparKculariteminadatabase– Performcomparisonsonthedrive– Returnsthedisksclosestmatches– Serverdoesfinalmerge
MediaMiningExample
• Performlowlevelimagetasksonthedisk!
• EdgeDetecKonperformedondisk– Senttoserverasedgeimage
– Serverdoeshigherlevelprocessing
WhynotjustuseabunchofPC’s?
• Theperformanceinceaseissimilar• Infact,thepaperessenKallyusedthissetuptoactuallybenchmarktheirresults!
• Supposedlythiscouldbecheaper• Thepaperdoesn’treallygiveagoodargumentforthis…– PossiblyreducedbandwidthondiskIOchannel– Butwhocares?
SomeQuesKons
• Whatcouldadiskpossiblydobeherthanthehostprocessor?
• WhataddedcostisassociatedwiththismediocreprocessorontheHDD?
• Arenewdependenciesareintroducedonhardwareandsosware?
• Perhapsother(beher)placestodothistypeoflocalparallelprocessing?
• Maybein2001thismademoresense?