On Grid Performance Evaluation using Synthetic Workloads
description
Transcript of On Grid Performance Evaluation using Synthetic Workloads
![Page 1: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/1.jpg)
April 22, 20231
University of Dortmund
On Grid Performance Evaluation using Synthetic Workloads
JSSPP 2006
Alexandru Iosup, Dick EpemaPDS Group, ST/EWI, TU Delft
Carsten Franke, Alexander Papaspyrou, Lars Schley, Baiyi Song, and Ramin Yahyapour UniDo
![Page 2: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/2.jpg)
April 22, 20232
University of Dortmund
Outline
• A Brief Introduction to Grid Computing• On Grid Performance Evaluation
Experimental Environments Performance Indicators General Workload Modeling Grid-Specific Workload Modeling The GrenchMark Framework
• Future Work• Conclusions
![Page 3: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/3.jpg)
April 22, 20233
University of Dortmund
A Brief Introduction to Grid Computing
• Typical grid environment• Applications [!]
• Unitary, composite• Data
• Resources• Compute (Clusters)• Storage• (Dedicated) Network
• Virtual Organizations, Projects• Groups, Users
• Grids vs. parallel production environments• Dynamic• Heterogeneous• Very large-scale (world)• No central administration→ Most resource management
problems are NP-hard
![Page 4: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/4.jpg)
April 22, 20234
University of Dortmund
Experimental Environments Real-World Testbeds
• Real-World Testbed• DAS, NorduGrid, Grid3/OSG, Grid’5000…
• Pros• True performance, also shows “it works!”• Infrastructure in place
• Cons• Time-intensive• Exclusive access (repeatability)• Controlled environment problem (limited scenarios)• Workload structure (little or no realistic data)• What to measure (new environment)
![Page 5: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/5.jpg)
April 22, 20235
University of Dortmund
Experimental Environments Simulated and Emulated Testbeds• Simulated and Emulated Testbeds• GridSim, SimGrid, GangSim, MicroGrid … • Essentially trade-off precision vs. speed
• Pros• Exclusive access (repeatability)• Controlled environment (unlimited scenarios)
• Cons• Synthetic Grids: What to generate? How to generate?
Clusters, Disks, Network, VOs, Groups, Users, Applications, etc.
• Workload structure (little or no realistic data)• What to measure (new environment)• Validity of results (accuracy vs. time)
![Page 6: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/6.jpg)
April 22, 20236
University of Dortmund
Grid Performance Evaluation Current Practice
• Performance Indicators• Define my own metrics, or use U and AWT/ART, or both
• Workload Structure• Run my own workload, or use traces that are not validated
by peer researchers; do not make comparisons!• Run benchmarks from typical parallel production
environments• Mostly all users are created equal assumptionNeed a common Need a common
performance performance evaluation framework evaluation framework
for Gridfor Grid
![Page 7: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/7.jpg)
April 22, 20237
University of Dortmund
Grid Performance Evaluation Current Issues
• Performance Indicators• What should be the metrics for the new
environment? • Workload Structure• Which general aspects could be important? • Which Grid-specific aspects need to be
addressed? Need a common Need a common performance performance
evaluation framework evaluation framework for Gridfor Grid
![Page 8: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/8.jpg)
April 22, 20238
University of Dortmund
Performance Indicators
• Time-, Resource-, and System-Related Metrics• Traditional: utilization, A(W)RT, A(W)WT, A(W)SD• New: waste, fairness (or service quality reliability)
• Workload Completion and Failure Metrics“ In Grids, functionality may be
even more important than performance ”• Workload Completion (WC)• Task and Enabled Task Completion (TC, ETC)• System Failure Factor (SFF)
![Page 9: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/9.jpg)
April 22, 20239
University of Dortmund
General Aspects for Workload Modeling• User/Group/VO model
• Detailed modeling for top-5/10 users, then clustering (Use squash area to group)
• Submission patterns• Yearly, monthly, weekly, daily• Do daily patterns exist? (Are Grids truly global?)
• Temporal patterns• Repeated submission (batches of jobs)• Job dependencies (composite applications common in Grid(?))
• Feedback • Empiric rules (don’t submit jobs when system busy). But,
reactive submission tools, co-allocators, evolving applications, etc.
![Page 10: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/10.jpg)
April 22, 202310
University of Dortmund
Grid-Specific Workload ModelingComputation Management
• Processor co-allocation• Fixed, non-fixed, semi-fixed jobs
• Job flexibility and composition• Moldable, evolvable, flexible, etc.• Batches, workflows, other dependecies
• Other aspects• Background load: define top jobs (by consumption),
model the rest as background load• Project stage
![Page 11: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/11.jpg)
April 22, 202311
University of Dortmund
Grid-Specific Workload ModelingData and Network Management
• Clearly Defined I/O Requirements• Files, streams, others• Data location and size
• Replicas• Replica location
• Other aspects• HDD occupancy
• Clearly Defined Network Requirements• Bandwidth, latency• Communication pattern
• Special Situations• Dedicated paths, other
QoS• Other aspects
• Background load
![Page 12: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/12.jpg)
April 22, 202312
University of Dortmund
Grid-Specific Workload ModelingLocality/Origin Management
• Job issuer and execution siteNot all VOs are created equal !
• Two-level view: Which VO generates the next job? Within a VO, which user generates the next job?
• Three-level view, Multi-level view (Project, VO, Group, User)• (Usage) Service Level Agreements
• Use my system 50% for 7 days, or 20% for 30 days• Dedicated paths, other QoS
• Other aspects• Background load pertaining to same (u)SLA
![Page 13: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/13.jpg)
April 22, 202313
University of Dortmund
Grid-Specific Workload ModelingFailure Modeling
• Error level• Infrastructure• Middleware• Application• User
• Fault tolerance scheme for submitted jobs• Catch the system feedback into the model
• Other aspects• Cascading errors
![Page 14: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/14.jpg)
April 22, 202314
University of Dortmund
Grid-Specific Workload ModelingEconomic Models
• Utility • Resource utility• Application utility
• Pricing policies • Time-dependent pricing: pay cheaper on off-peak hours• Load-dependent pricing: pay cheaper for unused resources• Package pricing: pay cheaper for bundles of resources• Trust-building pricing: pay cheaper as old users
• Other aspects• Available information• Penalty / user satisfaction
![Page 15: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/15.jpg)
April 22, 202315
University of Dortmund
GrenchMark: a Framework for Analyzing, Testing, and Comparing grids• What’s in a name?
grid benchmark → working towards a generic tool for the whole community: help standardizing the testing procedures, but benchmarks are too early; we use synthetic grid workloads instead
• What’s it about?A systematic approach to analyzing, testing, and comparing grid settings, based on synthetic workloads• A set of metrics for analyzing grid settings• A set of representative grid applications
• Both real and synthetic• Easy-to-use tools to create synthetic grid workloads• Flexible, extensible framework
![Page 16: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/16.jpg)
April 22, 202316
University of Dortmund
GrenchMark Overview: Easy to Generate and Run Synthetic Workloads
![Page 17: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/17.jpg)
April 22, 202317
University of Dortmund
… but More Complicated Than You Think• Workload structure
• User-defined and statistical models • Dynamic jobs arrival• Burstiness and self-similarity• Feedback, background load• Machine usage assumptions• Users, VOs
• Metrics• A(W) Run/Wait/Resp. Time • Efficiency, MakeSpan• Failure rate [!]
• (Grid) notions• Co-allocation, interactive jobs, malleable, moldable, …
• Measurement methods• Long workloads• Saturated / non-saturated system• Start-up, production, and cool-down scenarios• Scaling workload to system
• Applications• Synthetic• Real
• Workload definition language
• Base language layer• Extended language layer
• Other• Can use the same workload for both simulations and real environments
GrenchMark may become a vehicle for GrenchMark may become a vehicle for proving proving
(performance indicators, workload (performance indicators, workload modeling) modeling)
research in dynamic, heterogeneous, research in dynamic, heterogeneous, very large-scale environmentsvery large-scale environments
![Page 18: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/18.jpg)
April 22, 202318
University of Dortmund
GrenchMark: Iterative Research Roadmap
![Page 19: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/19.jpg)
April 22, 202319
University of Dortmund
GrenchMark: Iterative Research RoadmapSimple functional systemA.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema,
Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, CoreGRID IW, Nov 2005.
![Page 20: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/20.jpg)
April 22, 202320
University of Dortmund
GrenchMark: Iterative Research Roadmap
Open-GrenchMark
CommunityEffortThis work
Complex extensible systemA.Iosup, D.H.J.Epema, GrenchMark: A Framework for Analyzing,
Testing, and Comparing Grids, IEEE CCGrid'06, May 2006.
![Page 21: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/21.jpg)
April 22, 202321
University of Dortmund
• Performance Evaluation of Grid Systems - need a common performance evaluation framework for grids - need real grid traces (scheduling, accounting, monitoring, etc.) - need more research on workload modeling and performance indicators
• Performance indicators - failure metrics as important as traditional performance metrics
• Workload modeling - generic workload modeling needs validation based on real grid traces - computation/data/network management - locality/origin management - failure modeling - economic models
• GrenchMark - generic tool for the whole community - generates diverse grid workloads - easy-to-use, flexible, portable, extensible, …
Take home message
![Page 22: On Grid Performance Evaluation using Synthetic Workloads](https://reader035.fdocuments.net/reader035/viewer/2022070423/568167f7550346895ddd7266/html5/thumbnails/22.jpg)
April 22, 202322
University of Dortmund
Thank you!
Questions? Remarks? Observations? All welcome!
GrenchMark http://grenchmark.st.ewi.tudelft.nl/
http://http://grenchmark.st.ewi.tudelft.nl/grenchmark.st.ewi.tudelft.nl/