Benjamin Perry and Martin SwanyUniversity of Delaware

Computer Information Science

Background The problem The solution The results Conclusions and Future work

MPI programs communicate via MPI data types

MPI data types are usually modeled after native data types

Payloads are often arrays of MPI data types

The sending MPI library packs payload into contiguous block

The receiving MPI library unpacks payload into original form

Non-contiguous blocks incur a copy penalty SPMD programs, particularly in homogenous

environments, can use optimized packing

Background The problem The solution The results Conclusions and Future work

Users model MPI types after native types Some fields do not need to be transmitted Users often replace dead fields with a gap in

the MPI type to align with native type

Smaller payload…. but MPI type is non-contiguous

◦ Copy penalty during packing and unpacking Multi-core machines and high-performance

networks feel the cost depending on payload

Multi-core machines are becoming ubiquitous◦ SPMD applications are ideal for these platforms

Background The problem The solution The results Conclusions and Future work

Applies only to SPMD applications Static analysis to locate MPI data types

◦ MPI_type_struct() Build internal representation of MPI data

type ◦ MPI data type defined via library call at runtime◦ Parameters indicate base types, consecutive

instances, and displacements◦ Def/use analysis to determine static definition

Look for gaps in displacement array◦ Size of base types multiplied by consecutive array

Match MPI type to native type◦ Analyze the types of the payload◦ MPI type must be subset of native data structure◦ All sends and receives with MPI type handle must

also share same base types

Perform transformation on MPI type and native type◦ Adjust parameters in MPI_type_struct◦ Relocate non-transmitted fields to bottom of type

End goal: improve library performance of packing large arrays

Safety check◦ Cast to a type◦ Address-of

Except for computing displacement◦ Non-local types

Profitability◦ Sends / receives within loops◦ Large arrays of MPI types in sends / receives◦ Cost incurred by cache misses, locality by

adjusting native type when native type is in loops

Background The problem The solution The results Conclusions and Future work

LLVM compiler pass OpenMPI Intel Core2 Quad-core 2.4gz Ubuntu Control: sending un-optimized data type

with gap using payloads of various sizes Tested: Rearranging gap in MPI type and

native type using payloads of various sizes

Background The problem The solution The results Conclusions and Future work

MPI data types modeled after native data types

Users introduce gaps, making data noncontiguous and costly to pack on fast networks

Discover this scenario at compile time Fix it if safe and profitable Greatly improves multi-core performance;

infiniband also receives boost.

Data type fission with user-injected gaps◦ Separate transmitted fields from non-transmitted

fields◦ Complete eliminates data copy during packing

Data type fission with non-used fields◦ Perform analysis on receiving end to see which

fields are actually being used◦ Cull non-used fields from data type; perform

fission

? ? ? ?

? ?

? ? ?

?