SPEC Benchmarks for Large Systems
description
Transcript of SPEC Benchmarks for Large Systems
![Page 1: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/1.jpg)
Matthias Müller Höchstleistungsrechenzentrum Stuttgart
SPEC Benchmarks for Large Systems
Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann,
R. Lieberman, M. van Waveren, and B. Whitney
SPEC High Performance [email protected]
![Page 2: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/2.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
What is SPEC?
The Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops suites of benchmarks and also reviews and publishes submitted results from our member organizations and other benchmark licensees.
![Page 3: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/3.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC High-Performance Group
• Founded in 1994• Mission: To establish, maintain, and endorse a
suite of benchmarks that are representative of real-world high-performance computing applications.
• SPEC/HPG includes members from both industry and academia.
• Benchmark products:– SPEC OMP (OMPM2001, OMPL2001)– SPEC HPC2002 released at SC 2002
![Page 4: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/4.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Currently active SPEC HPG Members
• Fujitsu• HP• IBM• Intel• SGI• SUN• UNISYS• University of Purdue• University of Stuttgart
![Page 5: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/5.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Where is SPEC Relative to Other Benchmarks ?
There are many metrics, each one has its purposeRaw machine performance: TflopsMicrobenchmarks: StreamAlgorithmic benchmarks: LinpackCompact Apps/Kernels: NAS benchmarksApplication Suites: SPECUser-specific applications: Custom benchmarks
![Page 6: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/6.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Comparison of different benchmark classes
coverage relevance Identify problems
Time evolution
portability
Micro 0 0 ++ + +
Algorithmic - 0 + ++ +
Kernels 0 0 + + +
SPEC + + + + +
Apps - ++ 0 0 0
![Page 7: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/7.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMP
• Benchmark suite developed by SPEC HPG (High Performance Group)
• Benchmark suite for performance testing of shared memory processor systems
• Uses OpenMP versions of SPEC CPU2000 benchmarks and candidates
![Page 8: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/8.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMP/CPU2000 differences
• Larger working set sizes, 1.6GB for OMPM2001, 6.5 GB for OMPL2001; it is 200MB for CPU2000
• Longer run times (>1000 s/cpu for CPU2000 vs >10,000 s/cpu for OMP2000 medium)
• Focus on SMP systems, and issued by HPG• SPEC OMP based on work for CPU2000, SPEC
OMP mixes integer and FP in one suite• OMPM is focused on 4-way to 16-way systems• OMPL is targeting 32-way and larger systems
![Page 9: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/9.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMP Results
• 66 submitted results for OMPM• 24 submitted results for OMPL
Vendor HP HP SUN SGIArchitecture Superdome Superdome Fire 15K O3800
CPU PA-8700+ Itanium2 UltraSPARC III
R12000
Speed 875 1500 1200 400
L1 Inst 0.75 MB 16 KB 32 KB 32 KB
L1 Data 1.5 MB 16 KB 64 KB 32 KB
L2 - 256 KB 8 MB 8 MB
L3 - 6144 KB - -
![Page 10: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/10.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMPL Results: Benchmark with scaling to 128
![Page 11: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/11.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMPL Results: Superlinear scaling of applu
![Page 12: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/12.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMPL Results: Benchmark with scaling to 64
![Page 13: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/13.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC HPC2002 Benchmark
• Full Application benchmarks(including I/O) targeted at HPC platforms
• Serial and parallel (OpenMP and/or MPI)• Currently three applications:
– SPECenv: weather forecast – SPECseis: seismic processing, used
in the search for oil and gas– SPECchem: comp. chemistry, used
in chemical and pharmaceutical industries (gamess)
• All codes include several data sizes
![Page 14: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/14.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPECenv execution models on a Sun Fire 6800
Medium scales betterOpenMP best for small sizeMPI best for medium size
![Page 15: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/15.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC HPC2002 Results: SPECenv scaling
![Page 16: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/16.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPECseis execution models on a Sun Fire 6800
Medium scales betterOpenMP scales better than
MPI
![Page 17: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/17.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC HPC2002 Results: SPECseis scaling
![Page 18: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/18.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPECchem execution models on a Sun Fire 6800
Medium shows better scalability
MPI is better than OpenMP
![Page 19: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/19.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC HPC2002 Results: SPECchem scaling
![Page 20: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/20.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Hybrid Execution for SPECchem
![Page 21: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/21.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Current and Future Work
• SPEC HPC:– Update of SPECchem– Improving portability, including tools– Larger datasets
• New release of SPEC OMP:– Inclusion of alternative sources– Merge OMPM and OMPL on one CD
• Adoption of new benchmark codes:– Should represent a type of computation that is
regularly performed on HPC systems– We currently examine CPU2004 for candidates
![Page 22: SPEC Benchmarks for Large Systems](https://reader036.fdocuments.net/reader036/viewer/2022081513/56815dfa550346895dcc3379/html5/thumbnails/22.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Conclusion and Summary
• Results of OMPL and HPC2002:– Scalability of many programs to 128 CPUs
• Best choice of programming model (MPI,OpenMP, hybrid) depends on:– Hardware– Program– Data set size
• SPEC HPC will continue to update and improve the benchmark suites.