A General Framework for Algebraic Multigrid Method

Introduction Algebraic multigrid Framework Experiments Outlook References

A general software framework for algebraic multigrid methodswith applications to problems in reservoir simulations and CPR-AMG

Pavel Jiranek1

Serge Gratton1, Xavier Vasseur1, and Pascal Henon2

CERFACS, Toulouse, France1

Total SA, Centre Scientifique et Technique Jean-Feger, Pau, France2

International Conference on Preconditioning Techniquesfor Scientific and Industrial Applications

Oxford, UK

June 1921, 2013

Pavel Jiranek (CERFACS) A general software framework for algebraic multigrid methods PRECON13 1


Outline

1 Introduction

2 Algebraic multigrid

3 Framework

4 Experiments

5 Outlook



Outline

1 Introduction


3 Framework

4 Experiments

5 Outlook



Introduction

Simulations driven by partial differential equations require efficient methods for solving large andsparse systems of linear algebraic equations

Au = f ,

where A represents a discretised PDE or a system thereof.

Efficient methods should be scalable and robust.

Large sequences of systems often need to be solved, e.g., in time-dependent and/or nonlinearproblems such as these in reservoir simulations.

In order to satisfy the criteria of efficiency, iterative methods need a good preconditioner, whichshould be: 1) relatively cheap to compute and 2) substantially accelerate the convergence of theiteration.



Reservoir simulations

Multi-phase porous media flow models depending on the stage of the recovery process (single-,two-phase, compositional, thermal,...) are often discretised using a finite volume scheme in spaceand (full) implicit Euler scheme in time and linearised by a Newton type method: a system oflinear algebraic equations need to be solved at each Newton iteration within each time step.

The systems couple the pressure and saturation/concentration unknowns[App ApsAsp Ass

] [upus

]=

[fpfs

]and are nonsymmetric and ill-conditioned mainly due to the elliptic character of the pressureblock App and heterogeneity and anisotropy of the porous medium.

Example: SPE10 benchmark (Christie and Blunt [2001])

I cartesian grid with 60 220 85 106 cells,I highly heterogeneous and anisotropic (relative

variations of permeability 1011 and porosity 106).



Preconditioning techniques

Constrained pressure reduction (CPR)

Wallis et al. [1985], Lacroix et al. [2001], Scheichl et al. [2003]

I Transformation to a block triangular form (combining physical assumptions on the nonlocalpressure-saturation coupling and elimination of the local ones)[

App ApsAsp Ass

] [upus

]=

[fpfs

]

[App 0

Asp Ass

] [upus

]=

[fpfs

]

I Approximation of the diagonal blocks:

App exhibits the elliptic character and is usually approximated by a single cycle of algebraicmultigrid (AMG).

Ass is often strongly diagonally dominant and can be approximated, e.g., by an incompletefactorisation.

Motivation: In CPR-AMG, about 80% of the overall setup time is taken by the setup of the AMGpreconditioner for the block App (partial) reuse of the preconditioner.



Outline

1 Introduction


3 Framework

4 Experiments

5 Outlook



AMG basics

Algebraic vs geometric multigrid:

automatic coarsening without a priori knowledge of the geometry,

simple algebraic smoothers,

may be less efficient than the geometric multigrid,

setup phase.

Brandt et al. [1982], Ruge and Stuben [1987], Trottenberg et al. [2001],...

Setup phase:

I Computing the hierarchy by a recursive application of the setup procedure, which, for agiven A RNN , determines a coarse grid RNc , a prolongation operator P RNNc , arestriction operator R (= PT ), and the coarse grid matrix Ac (= RAP).

Solution phase:

I traditional cycles (V, W, F, ...) combining smoothing and coarse grid correction,

I polynomial acceleration in the coarse grid correction.Axelsson and Vassilevski [1989], Notay and Vassilevski [2008]



Ingredients of the setup phase

Coarsening: identifying a coarse grid and computing transfer operators (prolongation)

I Classical AMG-based methods: splitting variables to sets of F- and C-points.

I Aggregation-based AMG: partitioning variables to non-overlapping sets (aggregates).

P = P =

Graph coarsening: maximal independent set and partitioning algorithms on graph of Aassociated with strong couplings, e.g., classical AMG (Brandt et al. [1982], Ruge and Stuben[1987]), smoothed aggregation (Vanek et al. [1996]), compatible relaxation (Brandt [2000], Livne[2004]), energy-based coarsening (Brannick et al. [2006]).

Optimising convergence measures: AMGe (Brezina et al. [2000]), energy-minimising smoothing(Mandel et al. [1999]), AGMG (Napov and Notay [2012]),...

Adaptive methods: Brezina et al. [2004], Brezina et al. [2006],...



Ingredients of the setup phase

Parallel coarsening:

I Decoupled schemes running independently on each process (size of the coarse grid limitedby the number of processors, slow coarsening).

I Repartitioning to improve load balance on coarse grids possibly on a subset of processors(Chow et al. [2006]).

I Parallel coarsening schemes are based usually on modifications of parallel independent setalgorithms (Luby [1984], Jones and Plassman [1993], Adams [1998]) or on combinations ofdecoupled schemes with a special pre- or post-processing of the processor boundaryvariables, e.g., SAMG (Krechel and Stuben [2001]), BoomerAMG (Henson and Yang[2002]), ML (Tuminaro and Tong [2000]).



Outline

1 Introduction


3 Framework

4 Experiments

5 Outlook



Goals:

I object-oriented design and modularity,

I minimising coding requirements for both basic and advanced users,

I no fixed coarsening scheme (C/F splitting, aggregation) and problem type (scalar andcoupled problems, systems of PDEs),

I reuse of the hierarchy for preconditioner updates.

Basis: Boost, Trilinos (Epetra(Ext), Ifpack, Amesos, Aztec, Teuchos).



Design overview

Hierarchy contains

I a linked list of levels: operators A, R, P;

I cycle: computational part of AMG, (factories for) solvers.

Level variables:

I operator layer used for cycles: objects realising applications of A, R, P;

I variable layer used for setup: objects stored in a type-independent string-based dictionary;

I transfer between layers is made by variable transfers.

Setup of the hierarchy is made by a setup manager containing a set of components, which can beconsidered as blind parametrized black boxes with a set of inputs and outputs connected tocertain level variables.

The components are built in the order given by the data dependencies and priorities.

One setup manager can declare several variants: used to modify the behaviour depending on thedata reuse.



Example: Smoothed aggregation

Fine level Coarse level

A

Nullspace

I Declare the variable transfer.

I create the coupling graph

I aggregate on the graph

I create the nullspace approximations

I create the tentative prolongator

I smooth the tentative prolongator

I create the restrictor

I compute the coarse matrix

xx





A

Nullspace

I Declare the variable transfer and variant.

I create the coupling graph







xx

xx

xx





A

Nullspace

Graph

CouplingGraph

I Declare the variable transfer and variant,

I create the coupling graph according to achosen criterion, e.g., such that

(i, j) E(G) |aij | > |aiiajj |.







xx

xx

xxxx

xxxxxx

xxxxxx

xxxxxx

xxxxxxxx

xxxxxxxx

xxxxxxxx

xxxxxx

xxxx

xx





A

Nullspace

Graph Aggregator

Aggregates


I create the coupling graph,

I aggregate on the graph.






xx

xx

xxxx

xxxxxx

xxxxxx

xxxx

xx





A

Nullspace

Graph

NullspaceGenerator

Aggregates

NormalizedNullspace

Nullspace



I aggregate on the graph,

I create the nullspace approximations:

B [BT1 , . . . ,BTk ]T , Bi = QiRi.





xx

xx

xxxx

xxxxxx

xxxxxx

xxxxxx

xxxxxx

xxxxxx level="fine"

xxxxxx

xxxx

xx





A

Nullspace

Graph

TentativeProlongator

Aggregates

NormalizedNullspace

Nullspace

PTentative




I create the nullspace approximations,

I create the tentative prolongator:

P = blkdiag(Q1, . . . ,Qk).




xx

xx

xxxx

xxxxxx

xxxxxx

xxxxxx

xxxx

xx





A

Nullspace

Graph ProlongatorSmoother

Aggregates

NormalizedNullspace

Nullspace

PTentative

P





I create the tentative prolongator,

I smooth the tentative prolongator:

P = (I D1AF)P.



xx

xx

xxxx

xxxxxx

xxxxxx

xxxxxx

xxxxxx

xxxx

xx





A

Nullspace

Graph

TransposerAggregates

NormalizedNullspace

Nullspace

PTentative

P

R






I smooth the tentative prolongator,

I create the restrictor: R = PT .


xx

xx

xxxx

xxxxxx

xxxxxx

xxxx

xx





A

Nullspace

A

Graph

RAPProduct

Aggregates

NormalizedNullspace

Nullspace

PTentative

P

R







I create the restrictor,

I compute the coarse matrix: AC = RAP.

xx

xx

xxxx

xxxxxx

xxxxxx

xxxxxx

xxxxxx

xxxx

xx





A

Nullspace

A

Graph

Aggregates

NormalizedNullspace

Nullspace

PTentative

P

R







I create the restrictor,

I compute the coarse matrix.

xx

xx

xxxx...

xx




Management of the level variables and reusing the hierarchy

I normally, we keep all the variables created on levels during the setup;

I some variables can be declared as temporary: they are deleted as soon as they are notneeded anymore;

I reusing parts of the hierarchy can be achieved by clearing certain variables prior to setup orforcing to build certain components:

xx

xx

xxxx

xxxx

xxxx

xxxx

xx

Hierarchy h("sa.xml");

h[0].setOperator(A);

h.compute();

h.iterate(X, B);

...

h[0].setOperator(A new);

h.compute("reuse operators");

h.iterate(X new, B new);



Outline

1 Introduction


3 Framework

4 Experiments

5 Outlook



Parameters

I simple aggregation algorithm by Vanek et al. [1996] on a graph based on the symmetriccoupling criterion,

I local symmetric Gauss-Seidel as pre-, post-smoother, and coarse grid solver,

I size of the coarse grid < 16,

I GCG-based K-cycle,

I iterative solver: CG / GCR(10), relative residual tolerance 106,



Simple scalability experiments

BULL B510: 2 Intel Xeon Sandy Bridge @ 2.6 GHz (8 cores) & 64 GB memory per nodeFE discretisation of a 2D Poisson equation, 1800 grid points per node

Number of levels {8, . . . , 12}, number of iterations {15, . . . , 17}.



Reuse of the hierarchy

In CPR-AMG, we solve for each time-step a sequence of pressure systems for each Newtoniteration.

We consider six schemes of the reuse of the AMG preconditioner:

I S1: fixed preconditioner,

I S2: recomputed for each time-step,

I S3: recomputed for each time-step by reusing the transfer operators,

I S4: recomputed for each system by reusing the initial set of transfer operators,

I S5: recomputed for each time-step and updated by reusing the transfer operators,

I S6: recomputed from scratch for each system.



Reuse of the hierarchy

SPE10: Cumulative setup and solution time and iteration numbers(4 time-steps, 18 systems), L {8, 9}, Cop 1.4

Setup Solve Total NIT

S1 3.09E+000 5.48E+002 5.51E+002 1672

S2 1.34E+001 5.29E+001 6.63E+001 260

S3 6.91E+000 5.88E+001 6.57E+001 236

S4 2.61E+001 4.36E+001 6.97E+001 143

S5 3.51E+001 4.26E+001 7.77E+001 140

S6 6.49E+001 4.26E+001 1.08E+002 139



Outline

1 Introduction


3 Framework

4 Experiments

5 Outlook



Outlook

I optimisation,

I coupled coarsening schemes and classical AMG approaches,

I energy minimisation, adaptive methods,

I combining MPI with X and templating (Tpetra/Kokkos),

I coupled systems, e.g., saddle point problems.

CPR-related:

I coupled AMG approaches,

I Krylov subspace recycling and updates of the preconditioner.

Thank you for your attention!



References I

M. F. Adams. A parallel maximal independent set algorithm. In Proceedings of the 5th CopperMountain Conference on Iterative Methods, 1998.

O. Axelsson and P. S. Vassilevski. Variable-step multilevel preconditioning methods. i. Numer.Math., 56:157177, 1989.

A. Brandt. General highly accurate algebraic coarsening. Electron. Trans. Numer. Anal., 10:120,2000.

A. Brandt, S. F. McCormick, and J. W. Ruge. Algebraic multigrid (AMG) for automatic multigridsolutions with application to geodetic computations. Technical report, Institute forComputational Studies, Fort Collins, CO, 1982.

J. J. Brannick, M. Brezina, S. P. MacLachlan, T. A. Manteuffel, S. F. McCormick, and J. W.Ruge. An energy-based AMG coarsening strategy. Numer. Linear Algebra Appl., 13:133148,2006.

M. Brezina, A. J. Cleary, R. D. Falgout, V. E. Henson, J. E. Jones, T. A. Manteuffel, S. F.McCormick, and J. W. Ruge. Algebraic multigrid based on element interpolation (AMGe).SIAM J. Sci. Comput., 22:15701592, 2000.

M. Brezina, R. D. Falgout, S. P. MacLachlan, T. A. Manteuffel, S. F. McCormick, and J. W.Ruge. Adaptive smoothed aggregation (SA). SIAM J. Sci. Comput., 25(6):18961920, 2004.

M. Brezina, R. D. Falgout, S. P. MacLachlan, T. A. Manteuffel, S. F. McCormick, and J. W.Ruge. Adaptive algebraic multigrid. SIAM J. Sci. Comput., 27(4):12611286, 2006.



References II

E. Chow, R. D. Falgout, J. J. Hu, R. S. Tuminaro, and U. M. Yang. A survey of parallelizationtechniques for multigrid solvers. In M. A. Heroux, P. Raghavan, and H. D. Simon, editors,Parallel Processing in Scientific Computing, Software, Environments, Tools, pages 179201.SIAM, Philadelphia, PA, 2006.

M. A. Christie and M. J. Blunt. Tenth SPE comparative solution project: a comparison ofupscaling techniques. SPE Reservoir Engineering and Evaluation, 4:308317, 2001.

M. W. Gee, J. J. Hu, M. G. Sala, C. M. Seifert, and R. S. Tuminaro. ML 5.0 SmoothedAggregation Users Guide. Technical Report SAND2006-2649, Sandia National Laboratories,2007.

V. E. Henson and U. M. Yang. BoomerAMG: A parallel algebraic multigrid solver andpreconditioner. Appl. Numer. Math., 41:155177, 2002.

M. T. Jones and P. E. Plassman. A parallel graph coloring heuristic. SIAM J. Sci. Comput., 14:654669, 1993.

A. Krechel and K. Stuben. Parallel algebraic multigrid based on subdomain blocking. ParallelComput., 27:10091031, 2001.

S. Lacroix, Y. V. Vassilevski, and M. F. Wheeler. Decoupling preconditioners in the implicitparallel accurate reservoir simulator (IPARS). Numer. Linear Algebra Appl., 8:537549, 2001.

O. E. Livne. Coarsening by compatible relaxation. Numer. Linear Algebra Appl., 11(23):205227, 2004.



References III

M. Luby. A simple parallel algorithm for the maximal independent set problem. SIAM J.Comput., 15:10361053, 1984.

J. Mandel, M. Brezina, and P. Vanek. Energy optimization of algebraic multigrid bases.Computing, 62(3):205228, 1999.

A. Napov and Y. Notay. An algebraic multigrid method with guaranteed convergence rate. SIAMJ. Sci. Comput., 34(2):A1079A1109, 2012.

Y. Notay and P. S. Vassilevski. Recursive Krylov-based multigrid cycles. Numer. Linear AlgebraAppl., 15:473487, 2008.

J. W. Ruge and K. Stuben. Algebraic Multigrid. In S. F. McCormick, editor, Multigrid Methods,Frontiers in Applied Mathematics, pages 73130. SIAM, Philadelphia, PA, 1987.

R. Scheichl, R. Masson, and J. Wendebourg. Decoupling and block preconditioning forsedimentary basin simulations. Computat. Geosci., 7:295318, 2003.

U. Trottenberg, C. W. Oosterlee, and A. Schuller. Multigrid. Academic Press, London, UK, 2001.

R. S. Tuminaro and C. Tong. Parallel smoothed aggregation multigrid: aggregation strategies onmassively parallel machines. In J. Donnelley, editor, Supercomputing 2000 Proceedings, 2000.

P. Vanek, J. Mandel, and M. Brezina. Algebraic multigrid by smoothed aggregation for secondand fourth order elliptic problems. Computing, 56:179196, 1996.

J. R. Wallis, R. P. Kendal, and T. E. Little. Constrained residual acceleration of conjugateresidual methods. SPE Reservoir Simulation Symposium, SPE 13536, Dallas, TX, February1013, 1985.


IntroductionAlgebraic multigridFrameworkExperimentsOutlook

A General Framework for Algebraic Multigrid Method

Documents

Transcript of A General Framework for Algebraic Multigrid Method