Inferring the Topology and Traffic Load of Parallel Programs in a VM environment Ashish Gupta...
-
date post
21-Dec-2015 -
Category
Documents
-
view
214 -
download
0
Transcript of Inferring the Topology and Traffic Load of Parallel Programs in a VM environment Ashish Gupta...
Inferring the Topology and Traffic Load of Parallel Programs in a VM
environment
Ashish Gupta
Resource Virtualization
Winter Quarter Project
Motivation
• A distributed computing environment based on Virtual Machines
• Goal: Efficient execution of Parallel applications in such an environment
ParallelApplication Behavior
Intelligent Placement and virtual networking
of parallel applications
VM EncapsulationVirtual Networks
With VNET
Goal of this project
Low Level Traffic Monitoring
?
An online topology inference framework for a VM environment
Approach
Design an offline framework
Evaluate with parallel benchmarks
If successful, design an online framework for VMs
An offline topology inference framework
Goal:
A test-bed for traffic monitoring and evaluating topology inference
methods
The offline method
Synced Parallel Traffic Monitoring
Traffic Filtering and Matrix Generation
Matrix Analysis and Topology Characterization
The offline method
Synced Parallel Traffic Monitoring
Traffic Filtering and Matrix Generation
Matrix Analysis and Topology Characterization
h1 h2 h3 h4
h1 7.7 7.6 7.8
h2 13.1 6.6 6.5
h3 13.5 6.4 6.6
h4 13.2 6.5 6.5
*numbers indicate MB of data transferred.
The offline method
Synced Parallel Traffic Monitoring
Traffic Filtering and Matrix Generation
Matrix Analysis and Topology Characterization
Parallel Benchmarks used
• Synthetic benchmarks: Patterns– N-dimensional mesh-neighbor– N-dimensional toroid-neighbor– N-dimensional hypercubes– Tree reduction – All-to-All
• Scheduling mechanism to generate deadlock free and efficient schemes
1 2 3
Application benchmarks
• NAS PVM benchmarks– Popular benchmarks for parallel computing– 5 benchmarks
• PVM-POV : Distributed Ray Tracing
• Many others…
h1 h2 h3 h4 h5 h6 h7 h8
h1 19.0 19.6 19.2 19.6 18.8 13.7 19.3
h2 22.6 10.7 10.8 10.7 10.9 9.7 10.5
h3 22.2 8.78 11.2 10.4 10.1 10.5 10.5
h4 22.4 8.9 9.5 11.1 10.8 10.6 10.2
h5 22.3 10.0 9.51 9.72 11.7 10.9 11.9
h6 24.0 8.9 10.7 9.9 10.8 12.2 12.1
h7 23.2 10.0 9.7 9.5 10.3 10.2 12.0
h8 24.9 11.2 11.0 11.8 11.5 11.2 10.7 *numbers indicate MB of data transferred.
An Online Topology Inference Framework
Goal:
To automatically detect, monitor and report the global traffic matrix for a set of VMs running on a overlay network
Overall Design
• Extend VNET to include the required features– Allows a set of VMs to be on same Layer 2
domain– Monitoring at ethernet packet level
• Challenge– Lacks manual control– Detecting interesting parallel program
communication ?
Detecting interesting phenomenon
Reactive Mechanisms Proactive Mechanisms
•Certain address properties
•Based on Traffic rate
•Etc.
Provide support for queries by external agent
Rate based monitoring
Non-uniform discrete event sampling
What is the Traffic Matrix for the last n seconds ?
Traffic Analyzer
Rate based Change detection
Traffic MatrixQuery Agent
VM Network Scheduling Agent
VNET daemon
VM
VNET overlay network
To other VNET daemons
Physical Host
Traffic Matrix Aggregation
• Each VNET daemon keeps track of local traffic matrix– Need to aggregate this information for a global view– When the rate falls, the local daemons push the traffic
matrix
The proxy daemon