GENI Science Shakedown
description
Transcript of GENI Science Shakedown
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Motivation
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Motivation
Are the current GENI technologies capable of effectively supporting the computing needs of domain science?
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Motivation
Are the current GENI technologies capable of effectively supporting the computing needs of domain science?
What are the benefits and limitations of using GENI for domain science?
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Observations
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Observations
GENI has matured to the point where it is now time to evaluate its capabilities with respect to domain science applications “at scale” and use the results of this evaluation to inform the direction of future GENI development.
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Observations
GENI has matured to the point where it is now time to evaluate its capabilities with respect to domain science applications “at scale” and use the results of this evaluation to inform the direction of future GENI development.
There is an opportunity to introduce domain scientists to the GENI-way-of-thinking about networking and distributed systems in order to seed the dissemination of GENI technologies to existing and future high throughput and high-performance computing centers
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Personnel
• Computer Scientists:– Paul Ruth, [email protected]– Anirban Mandal, [email protected]
• Domain Scientists:– Jeffery L. Tilson, [email protected]– Brian Blanton, [email protected]
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Project Goals
• Apply the GENI experiment workflow to domain science applications and evaluate GENI’s ability to run domain science experiments
– Performance– Ease-of-use.
• Provide feedback to GENI rack developers on the current capabilities with respect to science applications as well as target areas for improvement.
• Build a tool for domain scientists to create RSpec from high-level descriptions of high-throughput and high-performance applications.
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Project Goals
• Apply the GENI experiment workflow to domain science applications and evaluate GENI’s ability to run domain science experiments (performance and ease-of-use).
• Build a tool for domain scientists to create RSpec from high-level descriptions of high-throughput and high-performance applications.
• Provide feedback to GENI rack developers on the current capabilities with respect to science applications as well as target areas for improvement.
Bonus Goal: Attract domain scientists to GENI
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Domain Science Applications
• MotifNetwork– Jeffery L. Tilson (RENCI)– High Throughput Computational
Genomics– Identification of functional domains
and the identification of conserved functional relationships across large numbers of genomes.
– Workflow ensemble (Pegasus/HTCondor)
– Task data sets on the order of 50-100 GB.
– Implicit iteration– Scales to 1000s of tasks
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Domain Science Applications
• ADCIRC– Brian Blanton (RENCI)
– Storm surge and tide model– Finite element model– MPI tightly coupled– Urgent Computing– Approved by FEMA for
computing storm surge flood hazard simulations
– Used for Digital Flood Insurance Rate Maps (DFIRMs)
– Scales to 10000+ MPI processes
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Experiments
• Provisioning overhead– creating, destroying, and modifying slices.
• GENI for big-data. – Large workflow inputs/output
• GENI for MPI. – Tightly coupled
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Outreach to Domain Science
• Tool for deploying domain science applications easily.
–RSpec generation of a Pegasus/HTCondor cluster and MPI cluster.
–Extendable to other science slice types.• Introducing domain scientists to GENI.• Feedback from domain scientists to inform future GENI development
15
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
Final thoughts on what we will find
• Most problems will be on the edges– Disk I/O
• Local disk• Network storage
– Network virtualization mechanism• Tradeoffs of: virtio, OVS, linux bridges, SR-IOV• More overhead than we realize
• Performance isolation• Something unexpected
16
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
First year goals
• GEC 19– Applications working on InstaGENI and ExoGENI– Initial performance profile
• GEC 20– Deeper performance profile– Initial suggestions for rack developers
17
The 18th GENI Engineering Conference October 27-29, 2013
Polytechnic Institute of New York University, Brooklyn, NY
ThanksQuestions?