Scalable line dynamics in ParaDiS - Stanford caiwei/papers/ParaDiS-SC04.pdf¢  ParaDiS...

download Scalable line dynamics in ParaDiS - Stanford caiwei/papers/ParaDiS-SC04.pdf¢  ParaDiS relies on spatial

If you can't read please download the document

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Scalable line dynamics in ParaDiS - Stanford caiwei/papers/ParaDiS-SC04.pdf¢  ParaDiS...

  • 0-7695-2153-3/04 $20.00 ©2004 IEEE

    Scalable line dynamics in ParaDiS

    Vasily Bulatov, Wei Cai, Jeff Fier, Masato Hiratani, Gregg Hommes, Tim Pierce, Meijie Tang, Moono Rhee, Kim Yates, Tom Arsenlis

    Lawrence Livermore National Laboratory1


    We describe an innovative highly parallel application program, ParaDiS, which computes the

    plastic strength of materials by tracing the evolution of dislocation lines over time. We discuss the

    issues of scaling the code to tens of thousands of processors, and present early scaling results of

    the code run on a prototype of the BlueGene/L supercomputer being developed by IBM in

    partnership with the US DOE’s ASC program.

    Introduction to dislocation dynamics

    The theory of crystal plasticity came into existence in the late nineteenth century [1]. With time it has become a proving ground for many novel methods of applied and computational mathematics. While very useful in practical applications in structural mechanics, geophysics, and other areas, mathematical crystal plasticity developed into a vestige of phenomenology, having little or, sometimes, no connection to the underlying microscopic physics of material behavior. Yet at the same time, metallurgists and physicists continued to ask hard questions: why do crystals deform in the ways they do and what are the mechanisms by which plasticity comes about?

    Realization that the plastic strength of crystalline materials is controlled by the motion of its line defects, dislocations, has firmed up in stages. In 1934, the concept of dislocations as physical agents of crystal plasticity was proposed, simultaneously and independently, by three eminent scientists – Taylor, Polanyi and Orowan. While essentially explaining most of the puzzling phenomenology of crystal plasticity, dislocations still remained only a beautiful hypothesis until the late 1950’s when first sightings of them were reported in electron microscopy experiments. Since then, the ubiquity and importance of dislocations for crystal plasticity and numerous other aspects of material behavior has been regarded as firmly established as, say, the role of DNA in promulgating life.

    Having largely answered why crystals deform as they do, dislocations provide a natural basis for bringing physical content into the mathematical theory of crystal plasticity. However, blocking the road to a grand unification of dislocation physics and continuum crystal plasticity is a very serious hurdle: in order to be representative of macroscopic plasticity behavior, the motion and interaction of dislocations en masse must be considered. Given that the physics of dislocation motion and interactions are very well understood, the nature of this obstacle is purely computational: one needs to be able to

    1 This work was performed under the auspices of the U.S. Department of Energy by the University of

    California at Lawrence Livermore National Laboratory under Contract No. W-7405-Eng-48. UCRL-


  • trace the simultaneous evolution of millions of dislocation lines over extended time intervals, in order to directly compute, as opposed to fit, the plastic strength of crystalline materials. This is one exemplary case where the size does matter. A mesoscopic approach of Dislocation Dynamics attempts to address this challenge.

    1µm 10µm

    (a) (b)

    Figure 1. (a) Dislocation microstructure developed in copper during plastic deformation, as observed in an electron microscope [2]. Dark regions contain large numbers of

    dislocation lines, whereas light regions contain no dislocations. (b) Dislocation

    microstructure developed during plastic deformation in a dislocation dynamics simulation by ParaDiS.

    Dislocation Dynamics: unifying dislocation physics and crystal plasticity

    The idea behind the computational approach of Dislocation Dynamics (DD) is simple. One introduces dislocation lines in the computational volume, letting them interact and move in response to forces inflicted by external loads and inter-dislocation interactions. The accuracy of DD simulations is assured by dislocation physics and mechanics that supply methods for computing forces acting on dislocations and dislocation velocities under these forces. Although the DD approach was introduced relatively recently, in the early 1990s, its potential impact on the science of material strength is well recognized [3- 5]. Yet, despite multiple pronouncements of inevitable success, the DD approach has not come close to the levels of computational performance required to demonstrate that it can, indeed, predict macroscopic strength from the collective behavior of dislocations. Given the computational complexity of the problem at hand, the limited success DD can claim so far is not surprising. At the same time, the performance levels required for “break-through” DD simulations can be very well quantified.

    Among various challenges worthy of large-scale computational attack, understanding the nature of strain hardening and dislocation patterning in metals is one of the most famous and holds a special value among researchers in the area of material strength. If direct DD simulations can be shown to accurately reproduce dynamic hardening transitions that

  • occur naturally during crystal deformation2, even skeptics will be convinced that the microscopic physical theory of crystal strength has arrived in the form of DD. With this in mind, we decided to gear our new DD code towards a single large-scale “hero” simulation that will cover the length and time scales sufficient to observe the hardening transitions that occur naturally as a result of collective motion and rearrangement of dislocations. Careful estimates show that to be able to naturally account for hardening and dislocation patterning and avoid “small volume” artifacts, the model should include from 1M to 100M dislocation segments. Furthermore, the evolution of such large dislocation groups will have to be traced over millions of time steps to reach the strain levels at which the hardening transitions are observed. DD capabilities available until now at LLNL and elsewhere stop short of these target performance figures by some 2-3 orders of magnitude. Massively parallel computing, such as will be available with BlueGene/L in 2005, is the only viable pathway to closing this performance gap.

    There are several challenges to this program that can be formulated at the outset. The first one is a programming challenge: DD models deal with the ever-changing topology of dislocation lines and line networks requiring rather complex data structures. On parallel machines, this challenge is dramatically amplified due to the need to handle the topology of these line networks across the boundaries of computational domains. The second challenge is a natural tendency of dislocation lines to cluster in space (owing to the long-range interactions among the lines) and develop highly heterogeneous distributions of degrees of freedom making it difficult to achieve a good load balance. The third challenge is to handle multiple time scales observed in the evolution of large dislocation ensembles, where periods of relative calm (slow motion of lines) are interspersed with bursts of very high activity over which small groups of lines move very fast and experience multiple collisions. The code ParaDiS recently written at LLNL has shown much promise in addressing these challenges.

    We have developed an interesting strategy for planning and executing simulations. In the course of deformation, dislocations multiply, increasing their numbers by 2-3 orders of magnitude. For this reason, it is possible to start with a relatively small model and let it grow on a small machine. Then, following a steadily growing number of dislocations, the job should be moved to progressively larger machines. This sort of progression worked very well when we scaled our simulation up from 12 to 100, then to 200 and, eventually, to 1,500 cpus. The same strategy will be applied on the Blue Gene/L machine, which is planned to have 131,072 processors when delivered to LLNL in early 2005. Scaling results for up to 4096 processors are shown in a later section of the paper.

    The ParaDiS project

    ParaDiS stands for Parallel Dislocation Simulator. It is a massively parallel dislocation dynamics simulation code that is being developed at the Lawrence Livermore National Laboratory since 2001. ParaDiS is specifically designed for investigating the collective behavior of large numbers of dislocation lines as required for understanding and accurate

    2 Strain hardening is responsible for some well-known facts of everyday life, such as why an aluminum

    paper clip eventually breaks after bending it back and forth several times.

  • prediction of plasticity and strength in crystals. The code is mainly written in C and uses the MPI library for communication among the processors.

    Figure 2. A fragment of the dislocation lines network: each line segment xy carries a unit

    of “vector current” quantified by Burgers vector bxy.

    The object of a ParaDiS simulation is a network of nodes connected by line segments. A node can have two or more segments, or arms, connecting it to neighboring nodes. Each segment carries a unit of “vector current” or Burgers vector, which denotes the direction and magnitude of the displacement that occurs when a dislocation moves. Similar to the scalar currents in electric circuits, Burgers vectors are conserved along the lines and satisfy the Kirchoff rule: the sum of Burgers vectors of all arms emanating from a given node is zero, e.g. b01 + b02 + b03 = 0 (