Application Skeletons: Constructing and Using Abstract...

www.ci.anl.govwww.ci.uchicago.edu


ApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience

DanielS.Katz1,AndreMerzky2,ZhaoZhang3,ShantenuJha2

1Computation Institute,UniversityofChicago&ArgonneNationalLaboratory2RADICALLaboratory,RutgersUniversity3AMPLab,UniversityofCalifornia,Berkeley

[email protected], andre.merzky.net, [email protected],[email protected]


2

Motivation

• Computer scientistswhobuildtoolsandsystemsneedtoworkonrealscientificapplications toprovetheeffectiveness oftheirtoolsandsystems– Andoftenvarythem– changeproblemsize,etc.

• However, accessingandbuilding realapplicationscanbehard(andisn’treally thecoreoftheirwork)– Someapplications(source)areprivatelyaccessible– Somedataisdifficulttoaccess– Someapplicationsuselegacycodeandaredependentonout-of-date

libraries– Someapplicationsarehardtounderstandwithoutdomainscience

expertise– Realapplicationsmaybedifficulttoscaleormodifytodemonstratesystem

trendsandcharacteristics



3

PartoftheAIMESProjectAIMES:AbstractionsandIntegrativeMiddlewareforExtremeScales• Goal

– Improveprinciplesandpracticeofdistributeddynamicresourcefederation

• Motivation– Identifyabstractions;implement tostudy&support federation– Improvedynamicanddistributedexecution onheterogeneous distributed

computinginfrastructure (DCI)o “Howwillmyapplicationperformon thisDCI?”o “HowcanIbestadaptmyapplication toaDCI?”o “HowcanthesetofresourcesDCIbestadapttomyapplication?”o “Whydid thesystemallocatethisDCItomyapplication?o “What“variables”mattermost…matterleast?

– Understand howdistributedworkloadexecution canbemanagedo Hypothesis: Usingmiddlewarethatsupports theintegrationofapplication-level

andresource-levelinformationo Questions:Whataretherelevantdecisions, andatwhatlevelshould theybe

made?



4

AIMESApproach• Useabstractionsthatcanbeflexiblycomposedandsupporta

rangeofexperiments,including– Skeletons representprimaryDCIapplication characteristics– ResourceBundles provide real-time infoonstateofdiverse

resources– PilotJobsenabledynamical

distributed resourcefederation&management

– Execution strategy:temporally orderedsetofdecisions thatneedtobemadewhenexecutingagivenworkload

• Experimenttounderstandrequirements&trade-offs



5

SkeletonTarget

• Wewanttobuildatoolsothat– Userscanquicklyandeasilyproduceasyntheticdistributedapplication

thatrepresents thekeydistributedcharacteristics ofarealapplicationo Thesyntheticapplicationshouldhavetasktype(serialorparallel),runtime,I/O

buffer,I/Oquantity,computationandI/Ointerleavingpatternandintertaskcommunicationthataresimilartothoseoftherealapplication

– Thesyntheticapplicationiseasytoruninadistributedenvironment:grids,clusters,andclouds

– Thesyntheticapplicationshouldbeexecutablewithcommondistributedcomputingmiddleware (e.g., SwiftandPegasus)aswellastheubiquitousUnixshell



6

ClassesofDistributedApplications

• BagofTasks:asetofindependenttasks• MapReduce:asetofdistributedapplicationwithkey-valuepairs

asintermediatedataformat• IterativeMapReduce:MapReduceapplicationwithiteration

requirement• Campaign:aniterativeapplicationwithavaryingsetoftasks

thatmustberuntocompletionineachiteration• Multi-stageWorkflow:asetofdistributedapplicationswith

multiplestagesthatusePOSIXfilesasintermediatedataformat• ConcurrentTasks:asetoftasksthathavetobeexecutedatthe

sametime(notsupportedbycurrentwork)

• Notethatmost/allofthesearemany-taskapplications



7

Contributions

• Anapplicationabstractionthatgivesusersgoodexpressivenessandeaseofprogrammingtocapturethekeyperformanceelementsofdistributedapplications

• AversatileSkeletontaskimplementationthatisconfigurable(numberoftasks,serialorparalleltasks,amountofI/Oandcomputation,I/Obuffersize,computationandI/Ointerleavingoptions)

• AninteroperableSkeletonimplementationthatworkswithmainstreamworkflowframeworksandsystems(Swift,Pegasus,andShell)

• TheusageofSkeletonapplicationstosimplifysystemoptimizationimplementationandhighlighttheirimpacts



8

Challenge

• Balance the ease of programming and usage with the performance gap between Skeleton applications and real applications

Eas

e of

Pro

gram

min

g

Performance DifferenceSmall Large

Eas

yH

ard

Real App

Skeleton App

Simple Model



9

AnMulti-StageApplicationExample

• Applicationshavestages• Eachstagehastasks

– Taskshavetypes(serial/parallel)– Taskshavecomputationlengths– Input/Output fileshavesizes– I/Oisthroughbuffers– Inputfilescanbe(pre)existingfilesor

Outputfilesfrompreviousstages– ComputationandI/Ocanbe

interleaved

• Eachstagehasinput/output files– Inputfilesmaptotaskswithpatterns



10

SkeletonAbstraction

• ApplicationSkeletonsabstractanapplicationusingatop-downapproach:anapplicationiscomposedofstages,eachofwhichiscomposedoftasks.

• Anapplicationcanbedefinedbyaconfigurationfilecontaining:– Numberofstages– Foreachstage

o Tasktypes(serial/parallel)o Tasks(numberandcomputationlength)o Numberofprocessesforeachtasko Inputfiles(number,sizes,andmappingtotasks)o Outputfiles(number,sizes)o I/Obuffersizeo ComputationandI/Ointerleavingoption



11

SkeletonToolDesign

• TheSkeletontoolisimplementedasaparser.

ConfigurationFile

SkeletonTool

PreparationScripts

Executables

Application(PegasusDAG,SwiftScript,

Shellcommands)



12

TaskExecutable

• AlltasksimplementedasonestandaloneCprogramviaparameters

• CprogramcanbecompiledasserialwithGCC,asparallelwithMPICCcompiler.

• Anexecutionexample:– task serial 1 5 65536 65536 1 1 0

Stage_1_Input/Stage_1_Input_0_1 Stage_1_Output/Stage_1_Output_0_1 4200000

– Path_to_Task Task_TypeNum_Processes Task_LengthRead_BufferWrite_BufferNum_InputNum_OutputInterleave_Option [Input_File][Output_FileOutput_Size]



13

ABagofTaskApplicationExample

1. Num_Stage =12.3. Stage_Name =Bag4. Task_Type =serial5. Num_Processes =16. Num_Tasks =47. Task_Length =uniform58. Read_Buffer =655369. Write_Buffer =6553610. Input_Files_Each_Task =111. Input_1.Source=filesystem12. Input_1.Size=uniform210000013. Output_Files_Each_Task =114. Output_1.Size=uniform420000015. Interleave_Option =0

https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/bag.inputApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience


14


• Otheroptions– Task_Type canbeparallel

o Task_Type =parallel– Task_Length canbeastatisticaldistribution

o Task_Length =normal [20,3]– Task_Length canbeapolynomialfunctionofinputfilesizeo Task_Length =polynomial [20,3]Input_1o 20*Input_1.Size^3

– OutputsizecanbeapolynomialfunctionofTask_Lengtho Ouptut_1.Size =polynomial [10,2]Lengtho 10*Length^2

– Interleaving_Option canbe...



15


• Interleaving_Optioncanbe

read sleep write

interleave-nothing:

interleave-read-compute:

interleave-compute-write:

interleave-all:

time

0

1

2

3



16

AMulti-stageWorkflow

• Num_Stage =3• Stage_Name =Stage_1• …• Stage_Name =Stage_2• …• Write_Buffer =65536• Input_Files_Each_Task =2• Input_Task_Mapping =combination

Stage_1.Output_12• …

• Stage_Name =Stage_3• …• Input_Files_Each_Task =6• Input_Task_Mapping =combination

Stage_2.Output_16• …

https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/multi-stage.input



17

Input_Task_Mapping

• SpecifyInput_Task_Mapping:– Input_Task_Mapping =combinationStage_1_output_12– Equivalentto“Nchoosek”mathematically– 4files,6tasks=>

o file0,file1:task0o file0,file2:task1o file0,file3:task2o file1,file2:task3o file1,file3:task4o file2,file3:task5

https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/multi-stage.inputApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience


18

Input_Task_Mapping

• IfInput_Task_Mapping isnotspecified– Input_Files_Each_Task =2– Input_1.Source =Stage_1.Output_1– Input_2.Source =Stage_1.Output_1

– Filesaremappedtotasksinanaturalorder– file0,file1:task0– file2,file3:task1– file4,file5:task2– file6,file7:task3



19

Input_Task_Mapping

• Externalmappingoption– Input_Task_Mapping =externalsample-input/mapping.sh– catsample-input/mapping.sh

o echoStage_1_Output_0_1 Stage_1_Output_0_2o echoStage_1_Output_0_1 Stage_1_Output_0_3o echoStage_1_Output_0_1 Stage_1_Output_0_4o echoStage_1_Output_0_2 Stage_1_Output_0_3o echoStage_1_Output_0_2 Stage_1_Output_0_4o echoStage_1_Output_0_3 Stage_1_Output_0_4

– Theith linemapstotheith task

https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/external-mapper.inputApplicationSkeletons:ConstructingandUsingAbstractManyTaskApplicationsineScience


20

ASingleStageIterativeApplication

Num_Stage =1

Stage_Name =Stage_1Task_Type =serialNum_Tasks =4Task_Length =uniform10Num_Processes =1Read_Buffer =65536Write_Buffer =65536Input_Files_Each_Task =1Input_1.Source=filesystemInput_1.Size=uniform1048576

Output_Files_Each_Task =1Output_1.Size=uniform1048576

Interleave_Option =0Iteration_Num =3Iteration_Stages =Stage_1Iteration_Substitute =Stage_1.Input_1,Stage_1.Output_1

https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/single-stage-iterative.input

Stage_1.Input_1andStage_1.Output_1 shouldhaveIDENTICALnumberoffiles



21

AMultiStageIterativeApplication

https://github.com/applicationskeleton/Skeleton/blob/master/src/sample-input/multiple-stage-iterative.input

Stage_Name =Stage_3Task_Type =serialNum_Tasks =6Task_Length =uniform32Num_Processes =1Read_Buffer =65536Write_Buffer =65536Input_Files_Each_Task =1Input_1.Source=Stage_2.Output_1


Interleave_Option =0Iteration_Num =3Iteration_Stages =Stage_3,Stage_4Iteration_Substitute =Stage_3.Input_1,

Stage_4.Output_1

Stage_Name =Stage_4Task_Type =serialNum_Tasks =6Task_Length =uniform32Num_Processes =1Read_Buffer =65536Write_Buffer =65536Input_Files_Each_Task =1Input_1.Source=Stage_3.Output_1


Interleave_Option =0

Stage_3.Input_1andStage_4.Output_1 shouldhaveIDENTICALnumberoffiles



22

DeterminingSkeletonParametersManually

• Steps– PlaceI/OfilesonRAMdisk,useUnixtimecommandtomeasureruntime

– UseUnixstrace commandtofindnumberofreadsandwrites,totaldatareadandwritten

– AlignI/Ocallsinsequenceorder1 todetermineI/Oconcurrency,I/Obuffersize,andinterleavingoption

1Z.Zhang,D.S.Katz,M.Wilde,J.Wozniak,I.Foster.MTCEnvelope:DefiningtheCapability ofLargeScaleComputers intheContextofParallelScriptingApplications, Proceedings of22ndInternational ACMSymposiumonHigh-PerformanceParallelandDistributedComputing(HPDC'13),2013



23

DeterminingSkeletonParametersAutomatically

• Wraptaskinaprofiler(usesLinuxperf,hardwareandkernelcounters,systemtools)– Measureresourceconsumptionovertimeasprofile:timedsequenceofCPUcycles,memoryanddiskI/Ooperations

– DependenciesbetweencomputeandI/Oimplicitlycapturedinprofileinorderofthesequence

• Profilesaresystemindependent– Assumingcomparableoptimizationatsoftwareandhardwarelevels

• Profilemetricsaresuitableasinputforemulation



24

SkeletonAppsvs.RealApps

• Applications:– Case1:a6x6degreeimagemosaicinMontage– Case2:thefirst256queriesofNRxNR testinBLAST– Case3:partialseismicstudyofCyberShakepostprocessingonsiteTest

• Platformconfiguration:– 64computenodesonIBMBlueGene/P– TasksarelaunchedwithAMFORA[1]– EachtaskstagesinputfilefromGPFS,executethetask,thenwritestheoutputfilestoGPFS

[1]ZhaoZhang,DanielS.Katz,TimothyG.Armstrong,JustinM.Wozniak,andIanT.Foster."Parallelizingtheexecutionofsequentialscripts.”ProceedingsoftheInternationalConferenceforHighPerformanceComputing,Networking,StorageandAnalysis(SC13).2013.



25

MontageStatistics

#Tasks

#Inputs

#Outputs

Input(MB)

Output(MB)

SkeletonTask Length(uniform)

InterleavingOption

Error inStage

mProject 1319 1319 2594 2800 10400 11.6 0 -0.2%

mImgtbl 1 1297 1 5200 0.8 30.1 0 -2.1%

mOverlaps 1 1 1 0.8 0.4 9.1 0 -0.2%

mDiffFit 3883 7766 7766 31000 487 1.8 0 -3.3%

mConcatFit 1 3883 1 1.1 4.3 2.1 0 -1.5%

mBgModel 1 2 1 4.5 0.07 288 0 0.03%

mBackground 1297 1297 1297 5200 5200 0.4 0 -1.6%

mAdd 1 1297 1 5200 7400 519 0 -0.9%

Total -1.3%



26

BLASTStatistics

#Tasks

#Inputs

#Outputs

Input(MB)

Output(MB)

SkeletonTask Length

InterleavingOption

Error inStage

split 1 1 64 3800 3800 0 3 -1.9%

formatdb 64 64 192 3800 4400 uniform42 3 -0.6%

blastp 1024 4096 1024 70402 966 normal[109.2,14.9]

3 1.6%

merge 16 1024 16 966 867 normal[4.4,4.1]

3 1.1%

Total 1.4%



27

CyberShake PostProcessing Statistics

#Tasks

#Inputs

#Outputs

Input(MB)

Output(MB)

SkeletonTask Length

InterleavingOption

Error inStage

Extract 128 130 256 5400 11000 uniform6.39

0 2.6%

Seis 4096 4352 4096 11000 96 normal[26.9,13.3]

0 2.4%

PeakGM 4096 4096 4096 96 1.4 uniform0.23

0 2.3%

Total 2.4%



28

UsingApplicationSkeletons

• DataCaching– Comparingsharedfilesystem(PVFS)andin-memoryfilesystem(AMFORA)performanceformProjectPP

– Using64n1-highmem-2instancesonGoogleComputeEngine(GCE)

PVFS AMFORA Improvement

mProjectPP-real 285.2seconds 100.9seconds 63.0%

mProjectPP-skeleton 273.7seconds 101.3seconds 64.6%



29


• TaskScheduling– Data-awareschedulingvs.FIFO– Using16n1-highmem-2GCEinstances

• mProjectPP-realhas0.7%improvement• mProjectPP-skeletonhas1.6%improvement

• Usedskeletonstoshow:5xlargerinputfilesize=>16.4%time-to-solutionimprovementwithdata-awarescheduling



30


• I/OTuning– Multiplemetadataservervs.Singlemetadataserver– Using16n1-highmem-2instances

• mProjectPP-realshows1.1%improvement• mProjectPP-skeletonshows1.2%improvement

• Useskeletonstoshow:10xshortertasklength=>31.2%improvementformultiplemetadataservers



31

RelatedWork

• Simplifyparallelapplicationcode– Extractkernels:NASParallelBenchmarks,BerkeleyDwarfs/Motifs,CORALbenchmarks,etc.

– Simplifynon-kernelpartofapp:Kerbyson (SC12),Worley(SC94),miniapps (e.g.,Mantevo,MADbench)

• Simplifyparallelapplicationsintime– Sodhi (Cluster2008)

• Usesystemtracesinplaceofapplications– Chen(VLDB12),Harter(FAST14),Ouserhout (NSDI15)

• Skeleton-likeapproaches– Skel (forI/O),Tigres (distributedapptemplates),WGL(similartoourwork,butsimpler)

• Otherworkthatusedthetermskeleton• SeeFGCSpaperforcomparisons



32

Conclusion• Skeleton toolcancompose skeleton applicationinatop-down

manner: application, stage,task• Skeleton taskabstractionallowsspecificationoftasktype,tasklength,

numberofprocesses, I/Obuffer, I/Oquantity, interleaving option,andfilenumber

• Cancreateeasy-to-access, easy-to-build, easy-to-change, andeasy-to-runbag-of-tasks, (iterative)map-reduce, and(iterative)multi-stageworkflow applications

• Skeletonapplications canbeeasily shared,makingmiddleware andtoolexperiments morereproducible

• Skeletonapplicationshaveperformance closetothatoftherealapplicationswithanoverall errorof-1.3%,1.5%,and2.4%forMontage,BLAST,andCyberShake PostProcessing

• Skeletons canshowtheeffectiveness ofsystemimprovements suchasdatacaching,taskscheduling, I/Otuning



33

FutureWork

• Useapplicationtracedatatoproduceskeletonapplications

• Determineawaytorepresentthecomputationalworkinataskthatwhencombinedwithaparticularplatformcangiveanaccurateruntimeforthattask

• Supportconcurrenttasksthatneedtorunatthesametimetoexchangeinformation

• Test ondistributedsystemswherelatencies,particularfileusage,andotherissuesmaybemoreimportantthanontheparallelsystemsandcloudenvironments



34

References• Software:

– TheSkeleton toolisopensourceat:https://github.com/applicationskeleton/Skeleton

– v1.2publishedas:D.S.Katz,A.Merzky,M.Turilli,M.Wilde,Z.Zhang2015.ApplicationSkeletonv1.2.,DOI:10.5281/zenodo.13750

– Tryit!Contribute toit!• Papers:

– Z.ZhangandD.S.Katz,"ApplicationSkeletons:EncapsulatingMTCApplicationTaskComputationandI/O,”Proceedings of6thWorkshoponMany-TaskComputingonGridsandSupercomputers (MTAGS), 2013.

– Z.Zhang,D.S.Katz,"UsingApplicationSkeletons toImproveeScienceInfrastructure," Proceedingsof10thIEEEInternational Conference oneScience,2014.DOI:10.1109/eScience.2014.9

– D.S.Katz,A.Merzky,Z.Zhang,S.Jha,"ApplicationSkeletons:ConstructionandUse ineScience," FutureGeneration ComputingSystems,2015.DOI:10.1016/j.future.2015.10.001



35

Acknowledgements

• ThisworkwassupportedinpartbytheU.S.DepartmentofEnergyundertheASCRawardDE-SC0008617(theAIMESproject)

• IthasbenefitedfromdiscussionswithMatteoTurilli,JonWeissman,andLavanya Ramakrishnan

• ComputingresourceswereprovidedbytheArgonneLeadershipComputingFacility

• WorkbyKatzwassupportedbytheNationalScienceFoundationwhileworkingattheFoundation.Anyopinion,finding,andconclusionsorrecommendationsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNationalScienceFoundation


Application Skeletons: Constructing and Using Abstract...

Documents

Transcript of Application Skeletons: Constructing and Using Abstract...