PETSc: 1001 Applicationspeople.cs.uchicago.edu/~knepley/presentations/KAUST...Free for everyone,...

PETSc: 1001 Applications

Matthew Knepley

Computation InstituteUniversity of Chicago

Department of Molecular Biology and PhysiologyRush University Medical Center

Department of Applied Mathematics and Computational ScienceKing Abdullah University of Science and Technology

Apr 5, 2010

M. Knepley (UC) PETSc KAUST 1 / 35

What can PETSc do?

Outline

1 What can PETSc do?What is PETSc?Who uses PETSc?

2 Value of Design


What can PETSc do? What is PETSc?

Outline




How did PETSc Originate?

PETSc was developed as a Platform forExperimentation

We want to experiment with differentModelsDiscretizationsSolversAlgorithms

which blur these boundaries


http://amzn.com/0521602866


The Role of PETSc

Developing parallel, nontrivial PDE solvers thatdeliver high performance is still difficult and re-quires months (or even years) of concentratedeffort.

PETSc is a toolkit that can ease these difficul-ties and reduce the development time, but it isnot a black-box PDE solver, nor a silver bullet.— Barry Smith


http://www.mcs.anl.gov/~bsmith


What is PETSc?

A freely available and supported research codeDownload from http://www.mcs.anl.gov/petscFree for everyone, including industrial usersHyperlinked manual, examples, and manual pages for all routinesHundreds of tutorial-style examplesSupport via email: [email protected] from C, C++, Fortran 77/90, and Python


http://www.mcs.anl.gov/petsc

mailto:[email protected]


What is PETSc?

Portable to any parallel system supporting MPI, including:Tightly coupled systems

Cray XT5, BG/Q, NVIDIA Fermi, Earth SimulatorLoosely coupled systems, such as networks of workstations

IBM, Mac, iPad/iPhone, PCs running Linux or Windows

PETSc HistoryBegun September 1991Over 60,000 downloads since 1995 (version 2)Currently 400 per month

PETSc Funding and SupportDepartment of Energy

SciDAC, MICS Program, AMR Program, INL Reactor ProgramNational Science Foundation

CIG, CISE, Multidisciplinary Challenge Program



The PETSc Team

Bill Gropp Barry Smith Satish Balay

Jed Brown Matt Knepley Lisandro Dalcin

Hong Zhang Victor Eijkhout Dmitry KarpeevM. Knepley (UC) PETSc KAUST 7 / 35

What can PETSc do? Who uses PETSc?

Outline




Who Uses PETSc?

Computational ScientistsEarth Science

PyLith (CIG)Underworld (Monash)Magma Dynamics (LDEO, Columbia)

Subsurface Flow and Porous MediaSTOMP (DOE)PFLOTRAN (DOE)

CFDOpenFOAMfreeCFDOpenFVM

MicroMagnetics (MagPar)Fusion (NIMROD)


http://www.geodynamics.org/cig/software/pylith

http://www.underworldproject.org/

http://stomp.pnnl.gov/

http://ees.lanl.gov/pflotran/

http://www.openfoam.com/

http://www.freecfd.com/

http://openfvm.sourceforge.net/

http://www.magpar.net/

http://www.nimrodteam.org/


Who Uses PETSc?

Algorithm DevelopersFinite Elements

PETSc-FEMlibMeshDeal IIOOFEM

Iterative methodsDeflated GMRESLGMRESQCGSpecEst

Preconditioning researchersPrometheus (Adams)ParPre (Eijkhout)FETI-DP (Klawonn and Rheinbach)

Fast Multipole Method (PetFMM)Radial Basis Function Interpolation (PetRBF)Eigensolvers (SLEPc)Optimization (TAO)


http://www.cimec.org.ar/petscfem

http://libmesh.sourceforge.net/

http://www.dealii.org/

http://www.oofem.org/

http://www.columbia.edu/~ma2325/prom_intro.html

http://www.netlib.org/scalapack/manual.ps

http://www.uni-due.de/numerik/klawonn.shtml

http://www.uni-due.de/numerik/rheinbach.shtml

http://barbagroup.bu.edu/Barba_group/PetFMM.html

http://barbagroup.bu.edu/Barba_group/PetRBF.html

http://www.grycap.upv.es/slepc/

http://www.mcs.anl.gov/tao


What Can We Handle?

PETSc has run implicit problems with over 500 billion unknownsUNIC on BG/P and XT5PFLOTRAN for flow in porous media

PETSc has run on over 224,000 cores efficientlyUNIC on the IBM BG/P Intrepid at ANLPFLOTRAN on the Cray XT5 Jaguar at ORNL

PETSc applications have run at 22 TeraflopsKaushik on XT5LANL PFLOTRAN code


http://www-unix.mcs.anl.gov/petsc/petsc-as/publications/petscapps.html#scalinglarge


PyLith

Multiple problemsDynamic ruptureQuasi-static relaxation

Multiple modelsNonlinear visco-plasticFinite deformationFault constitutivemodels

Multiple meshes1D, 2D, 3DHex and tet meshes

ParallelPETSc solversSieve meshmanagement

a

aAagaard, Knepley, Williams



Multiple Mesh Types

Triangular Tetrahedral

Rectangular Hexahedral



Magma Dynamics

Couples scalesSubductionMagma Migration

PhysicsIncompressible fluidPorous solidVariable porosity

Deforming matrixCompaction pressure

Code generationFEniCS

Multiphysics PreconditioningPETSc FieldSplit

a

aKatz, Speigelman



Fracture Mechanics

Full variational formulationPhase fieldLinear or Quadratic penalty

Uses TAO optimizationNecessary for linear penaltyBacktacking

No prescribed cracksArbitrary crack geometryArbitrary intersections

Multiple materialsComposite toughness a

aBourdin



Fracture Mechanics

1

1BourdinM. Knepley (UC) PETSc KAUST 15 / 35


Vortex Methodt = 000

Incompressible FlowGaussian vortex blobsHigh Re

PetFMM2D/3D domainsAutomatic load balancingVariety of kernelsOptimized with templates

PetRBFVariety of RBFsUses PETSc solversScalable preconditioner

ParallelismMPIGPU

a

aCruz, Yokota, Barba, Knepley







ParallelismMPIGPU

a




FEniCS-AppsRheagen

RheologiesMaxwellGrade 2Oldroyd-B

StabilizationDGSUPGEVSSDEVSSMacroelement

AutomationFIAT (elements)FFC (weak forms)

a

aTerrel



FEniCS-AppsRheagen

RheologiesMaxwellGrade 2Oldroyd-B

StabilizationDGSUPGEVSSDEVSSMacroelement

AutomationFIAT (elements)FFC (weak forms)

a

aTerrelM. Knepley (UC) PETSc KAUST 17 / 35


Real-time Surgery

Brain SurgeryElastic deformationOverlaid on MRIGuides surgeon

Laser Thermal TherapyPDE constrainedoptimizationPer-patient calibrationThermal inverse problem a

aWarfield, Ferrant, et.al.



Real-time Surgery

Brain SurgeryElastic deformationOverlaid on MRIGuides surgeon

Laser Thermal TherapyPDE constrainedoptimizationPer-patient calibrationThermal inverse problem

frastructure [1, 6] inherent to the control system relies critically on the precise real-time orchestration of

large-scale parallel computing, high-speed data transfer, a diode laser, dynamic imaging, visualizations,

inverse-analysis algorithms, registration, and mesh generation. We demonstrated that this integrated tech-

nology has significant potential to facilitate a reliable minimally invasive treatment modality that delivers

a precise, predictable and controllable thermal dose prescribed by oncologists and surgeons. However, MR

guided LITT (MRgLITT) has just recently entered into patient use [4] and substantial translational research

and validation is needed to fully realize the potential of this technology [20, 23] within a clinical setting. The

natural progression of the computer driven MRgLITT technology will begin with prospective pre-treatment

planning. Future innovations on the delivery side will likely involve combining robotic manipulation of fiber

location within the applicator as well as multiple treatment applicators firing simultaneously.

2D Slice

Catheter Entry

Prostate

Thermal Field

Skin

Figure 1: 3D volume rendering of in vivo MR-guided LITT delivery in a canine model of prostate. Contrastenhanced T1-W MR images have been volume rendered to better visualize the relationship of the targetvolume and applicator trajectory to the surrounding anatomy.. As displayed, the subject was stabilized inthe supine position with legs upward. A stainless steel stylet was used to insert the laser catheter consistingof a 700 µm core diameter, 1 cm di!using-tip silica fiber within a 2mm diameter water-cooled catheter (lightblue cylinder). A volume rendering of the multi-planar thermal images (in degrees Celsius) is registered andfused with the 3D anatomy to visualize the 3D volume of therapy while an axial slice cut from the principletreatment plane demonstrates a 2D representation of the local heating in that slice. The full field of viewshown is 240mm x 240mm (scale on image in mm).

2

a

aFuentes, Oden, et.al.


Value of Design

Outline

1 What can PETSc do?

2 Value of DesignDAMeshDMMGPCFieldSplit


Value of Design

Main Point

Separating theLocal from Global,

simplifies design,and enables modern solvers.


Value of Design

Global and Local

Local (analytical)Discretization/Approximation

FEM integralsFV fluxes

Boundary conditionsLargely dim dependent(e.g. quadrature)

Global (topological)Data management

Sections (local pieces)Completions (assembly)

Boundary definitionMultiple meshes

Mesh hierarchies

Largely dim independent(e.g. mesh traversal)


Value of Design DA

Outline



Value of Design DA

Ghost Values

To evaluate a local function f (x), each process requiresits local portion of the vector xits ghost values, bordering portions of x owned by neighboringprocesses

Local NodeGhost Node


Value of Design DA

DA Paradigm

The DA interface is based upon local callback functionsFormFunctionLocal(), set by DASetLocalFunction()

FormJacobianLocal(), set by DASetLocalJacobian()

When PETSc needs to evaluate the nonlinear residual F (x),Each process evaluates the local residual

PETSc assembles the global residual automaticallyUses DALocalToGlobal() method


Value of Design DA

Bratu Residual Evaluation

∆u + λeu = 0

ResLocal(DALocalInfo *info,Field **x,Field **f,void *ctx){/* Not Shown: Handle boundaries *//* Compute over the interior points */for(j = info->ys; j < info->ys+info->ym; ++j) {

for(i = info->xs; i < info->xs+info->xm; ++i) {u = x[j][i];u_xx = (2.0*u - x[j][i-1] - x[j][i+1])*hydhx;u_yy = (2.0*u - x[j-1][i] - x[j+1][i])*hxdhy;f[j][i] = u_xx + u_yy - hx*hy*lambda*exp(u);

}}

}

$PETCS_DIR/src/snes/examples/tutorials/ex5.cM. Knepley (UC) PETSc KAUST 25 / 35

Value of Design Mesh

Outline




Mesh Paradigm

The Mesh interface also uses local callback functionsmaps between global Vec and local Vec

Local vectors are combined into a Section object

When PETSc needs to evaluate the nonlinear residual F (x),Each process evaluates the local residual for each element

PETSc assembles the global residual automaticallySectionComplete() generalizes DALocalToGlobal()



Ghost Values

To evaluate a local function f (x), each process requiresits local portion of the vector xits ghost values, bordering portions of x owned by neighboringprocesses

Local NodeGhost Node



Integration

cells = mesh->heightStratum(0);for(c = cells->begin(); c != cells->end(); ++c) {<Compute cell geometry><Retrieve values from input vector>for(q = 0; q < numQuadPoints; ++q) {<Transform coordinates>for(f = 0; f < numBasisFuncs; ++f) {<Constant term><Linear term><Nonlinear term>elemVec[f] *= weight[q]*detJ;

}}<Update output vector>

}<Aggregate updates>



Integration

cells = mesh->heightStratum(0);for(c = cells->begin(); c != cells->end(); ++c) {coords = mesh->restrict(coordinates, c);v0, J, invJ, detJ = computeGeometry(coords);<Retrieve values from input vector>for(q = 0; q < numQuadPoints; ++q) {<Transform coordinates>for(f = 0; f < numBasisFuncs; ++f) {<Constant term><Linear term><Nonlinear term>elemVec[f] *= weight[q]*detJ;


}<Aggregate updates>M. Knepley (UC) PETSc KAUST 29 / 35


Integration






Integration

cells = mesh->heightStratum(0);for(c = cells->begin(); c != cells->end(); ++c) {<Compute cell geometry>inputVec = mesh->restrict(U, c);for(q = 0; q < numQuadPoints; ++q) {<Transform coordinates>for(f = 0; f < numBasisFuncs; ++f) {<Constant term><Linear term><Nonlinear term>elemVec[f] *= weight[q]*detJ;





Integration






Integration

cells = mesh->heightStratum(0);for(c = cells->begin(); c != cells->end(); ++c) {<Compute cell geometry><Retrieve values from input vector>for(q = 0; q < numQuadPoints; ++q) {realCoords = J*refCoords[q] + v0;for(f = 0; f < numBasisFuncs; ++f) {<Constant term><Linear term><Nonlinear term>elemVec[f] *= weight[q]*detJ;





Integration






Integration

cells = mesh->heightStratum(0);for(c = cells->begin(); c != cells->end(); ++c) {<Compute cell geometry><Retrieve values from input vector>for(q = 0; q < numQuadPoints; ++q) {<Transform coordinates>for(f = 0; f < numBasisFuncs; ++f) {elemVec[f] += basis[q,f]*rhsFunc(realCoords);<Linear term><Nonlinear term>elemVec[f] *= weight[q]*detJ;





Integration






Integration

cells = mesh->heightStratum(0);for(c = cells->begin(); c != cells->end(); ++c) {<Compute cell geometry><Retrieve values from input vector>for(q = 0; q < numQuadPoints; ++q) {<Transform coordinates>for(f = 0; f < numBasisFuncs; ++f) {<Constant term>for(d = 0; d < dim; ++d)for(e) testDerReal[d] += invJ[e,d]*basisDer[q,f,e];

for(g = 0; g < numBasisFuncs; ++g) {for(d = 0; d < dim; ++d)for(e) basisDerReal[d] += invJ[e,d]*basisDer[q,g,e]elemMat[f,g] += testDerReal[d]*basisDerReal[d]

elemVec[f] += elemMat[f,g]*inputVec[g];}<Nonlinear term>elemVec[f] *= weight[q]*detJ;





Integration






Integration

cells = mesh->heightStratum(0);for(c = cells->begin(); c != cells->end(); ++c) {<Compute cell geometry><Retrieve values from input vector>for(q = 0; q < numQuadPoints; ++q) {<Transform coordinates>for(f = 0; f < numBasisFuncs; ++f) {<Constant term><Linear term>elemVec[f] += basis[q,f]*lambda*exp(inputVec[f]);elemVec[f] *= weight[q]*detJ;





Integration






Integration


}}mesh->updateAdd(F, c, elemVec);




Integration






Integration



}Distribution<Mesh>::completeSection(mesh, F);


Value of Design DMMG

Outline




DMMG Paradigm

The DMMG interface uses the local DA/Mesh callback functions toassemble global functions/operators from local pieces

assemble functions/operators on coarse grids

DMMG relies upon DA/Mesh (DM) to organize theassembly

coarsening/refinementwhile it organizes the control flow for the multilevel solve.



Multigrid with DMMG

Allows multigrid with some simple command line options

-dmmg_nlevels

-pc_mg_type, -pc_mg_cycle_type-mg_levels_1_ksp_type, -mg_levels_1_pc_type-mg_coarse_ksp_type, -mg_coarse_pc_type-dmmg_view

Interface also works with 3rd party packages, like ML from Sandia


Value of Design PCFieldSplit

Outline




MultiPhysics Paradigm

The PCFieldSplit interfaceextracts functions/operators corresponding to each physics

VecScatter and MatGetSubMatrix() for efficiency

assemble functions/operators over all physicsGeneralizes LocalToGlobal() mapping

is composable with ANY PETSc solver and preconditionerThis can be done recursively

FieldSplit provides the buildings blocksfor multiphysics preconditioning.



MultiPhysics Paradigm

The PCFieldSplit interfaceextracts functions/operators corresponding to each physics

VecScatter and MatGetSubMatrix() for efficiency

assemble functions/operators over all physicsGeneralizes LocalToGlobal() mapping

is composable with ANY PETSc solver and preconditionerThis can be done recursively

Notice that this works in exactly the same manner asmultiple resolutions (MG, FMM, Wavelets)

multiple domains (Domain Decomposition)

multiple dimensions (ADI)



Stokes Example

The common block preconditioners for Stokes require only options:

-pc_type fieldsplit

-pc_field_split_type

-fieldsplit_0_pc_type ml

-fieldsplit_0_ksp_type preonly

(A B

BT 0

)



Stokes Example


-pc_type fieldsplit

-pc_field_split_type additive



-fieldsplit_1_pc_type jacobi


(A 00 I

)



Stokes Example


-pc_type fieldsplit

-pc_field_split_type multiplicative



-fieldsplit_1_pc_type jacobi


(A B0 I

)



Stokes Example


-pc_type fieldsplit

-pc_field_split_type schur



-fieldsplit_1_pc_type none

-fieldsplit_1_ksp_type minres

(A 00 −S

)-pc_fieldsplit_schur_factorization_type diag



Stokes Example


-pc_type fieldsplit






(A 0

BT S

)-pc_fieldsplit_schur_factorization_type lower



Stokes Example


-pc_type fieldsplit






(A B0 S

)-pc_fieldsplit_schur_factorization_type upper



Stokes Example


-pc_type fieldsplit


-pc_fieldsplit_schur_factorization_type full

(I 0

BT A−1 I

)(A 00 S

)(I A−1B0 I

)M. Knepley (UC) PETSc KAUST 35 / 35

Conclusions

Conclusion

With good code design, PETSc can makeLinear Algebra


Conclusions

Conclusion

With good code design, PETSc can makeSolving Equations


Conclusions

Conclusion

With good code design, PETSc can makeMesh and Data Management


Conclusions

Conclusion

With good code design, PETSc can makeMultigrid


Conclusions

Conclusion

With good code design, PETSc can makeMultiphysics


Conclusions

Conclusion

With good code design, PETSc can makeFMM


Conclusions

Conclusion

With good code design, PETSc can makeScientific Computing


PETSc: 1001 Applicationspeople.cs.uchicago.edu/~knepley/presentations/KAUST...Free for everyone,...

Documents

Transcript of PETSc: 1001 Applicationspeople.cs.uchicago.edu/~knepley/presentations/KAUST...Free for everyone,...