LINEAR SOLVERS FOR RESERVOIR

SIMULATION

CERFACS

Serge Gratton, Pavel Jiranek, Xavier Vasseur

INRIA-LJLL

Hussam Al Daas, Laura Grigori

TOTAL E&P

Pascal Hénon

SUMMARY

CEMRACS 2016

● Introduction

● Works on AMG for reservoir (collaboration with CERFACS)

● PhD 2015-2017: Enlarge GMRES (INRIA: H. Al Daas, L. Grigori)

● Conclusions and Prospects

2

INTRODUCTION

CEMRACS 2016 3

RESERVOIR SIMULATION : PURPOSES

4

● Estimation of Recovery Factor, Production plateau, EOR, modelling physic

● We have a limited knowledge of underground properties (fault, kr, ..)

● A big part of the work consists in « history matching » : needs many runs

CEMRACS 2016

DYNAMIC SIMULATOR MAIN EQUATIONS TO BE SOLVED

Conclusion:

For each cell and for each component, convergence is evaluated by measuring

the non-linear residual Rfl

5 Advanced Reservoir Simulation - Dec 2014 5

LINEAR EQUATION RESOLUTION

Conclusion: Most of the CPU is spent on linear resolution

6 Advanced Reservoir Simulation - Dec 2014 6

PARALLEL PERFORMANCE

● Scalability of a few models : (upper one =13M cells BO: around 50k

cells/processors)

7

Source : Schlumberger : SPE ACTE, Oct 31th 2011

CEMRACS 2016

SCALABILITY BOTTLENECK IN RESERVOIR SIMULATION

● Distributed memory framework (MPI)

● Load balancing : work per reservoir cell varies during the simulation.

● Linear solver method CPR-AMG is scalable with number of unknowns (weak

scalability) but poorly scalable with number of processors (strong scalability)

8

CEMR

ACS

2016

0

10

20

30

40

50

60

IX

Property calc

Linearizer

Linear solver

Updater

Boundary cells

B Interior cells

Ghost cells

● The Constraint Pressure Residual (CPR) solver : this solver specific to reservoir is a

two stages method (John Wallis and co. SPE 1985) :

- M1 : 1er stage is global : find approximate pressure (near-elliptic problem : AMG)

- M2 : 2nd stage is applied on the ARR system (eg BILU(0)) : block preconditioner

LINEAR SOLVER : CONSTRAINT PRESSURE RESIDUAL

PPPSPW

SPSSSW

WPWSWW

AAA

AAA

AAA

PPPS

SPSS

RR

t

AA

AAPAP ..

RRRW

WRWW

AA

AA

WWA

RWA

WPAWWA

RWA

WPA

1

1

1

1

1

2

1 ).( MMAIMMCPR

9 CEMRACS 2016

IMPROVING AMG FOR RESERVOIR

SIMULATION

CEMRACS 2016 10

CEMRACS 2016

RESERVOIR SIMULATION: ONE ITERATION OF CPR-AMG

1

1

1

1

1

2

1 ).( MMAIMMCPR

Pressure pseudo-

decoupling

11

The critical performance bottleneck in reservoir is the pressure solver part

CPR

AMG update

(different grid

levels)

SCALABILITY IN A RESERVOIR SIMULATION

CEMRACS 2016 12

● Example on a one million grid blocks BO model (SPE10) in TPP

● The figure represents TCPU*number of procs (perfect == same

length)

● The scalability bottleneck is clearly in AMG (setup + solve)

0,00E+00

2,00E+02

4,00E+02

6,00E+02

8,00E+02

1,00E+03

1,20E+03

1,40E+03

1,60E+03

1,80E+03

2,00E+03

1 2 4 8 16 32 64 128

CP

U t

ime (

s)

Number of MPI process (x8 = Nb cores)

AMG Solve

AMG Setup

CPR

Reservoir

● Pavel Jiranek : post-doc CERFACS 2012 – 2014 then 15 month in CSTJF until 11-30-2015

● Hybrid strategy Aggregation/RS is more efficient than standard RS strategy

● PMG package relies on MPI/Thread parallelization

● Ref: « Reducing complexity of algebraic multigrid by aggregation », in Num. Lin. Algebra.

with applications

● AMG based on Aggregation is efficient for the two first levels

PMG: HYBRID AGGREGATION/CLASSIC COARSENING

13 CEMRACS 2016

Classic coarsening Aggregation coarsening

RESULTS: PMG VS BOOMERAMG VS SAMG

CEMRACS 2016 14

● BoomerAMG from Lawrence Livermore (DOE).

● SAMG from Fraunhofer Institute.

● Minimum setup consists in keeping same grid interpolators for

several linear system (we limit to the same time step)

● On small-medium cases (< 1’000’000 cells); no significant

improvement over BoomerAMG for total simulation time.

AMG CPU Time

x1.31

RESULTS: PMG VS BOOMERAMG (MPI+THREAD)

CEMRACS 2016 15

● Viscous fingering study.

● Tuned parameter for BoomerAMG: aggressive coarsening

● 60’000’000 cells model. Test up to 8192 cores (1024 processors of

PANGEA1).

● The speed-up is more important (first layers of AMG represents more

computations).

x5

x2

1024*8 = 8192 cores

ENLARGED KRYLOV METHOD

CEMRACS 2016 16

Slides courtesy of Laura Grigori and Hussam Al Daas

GMRES: SCALABILITY LIMITATION

CEMRACS 2016 17

ENLARGE KRYLOV METHOD (L. GRIGORI ET AL 2014)

CEMRACS 2016 18

ENLARGED GMRES: ALGORITHM

CEMRACS 2016 19

● TSQR = Tall Skinny QR

ENLARGED GMRES: DETAILS

CEMRACS 2016 20

ENLARGED GMRES: EXPERIMENTS (1/2)

CEMRACS 2016 21

ENLARGED GMRES: EXPERIMENTS (2/2)

CEMRACS 2016 22

CONCLUSION & PROSPECTS

CEMRACS 2016 23

CONCLUSION & PROSPECTS

CEMRACS 2016 24

● Works on AMG: some improvements for large models in parallel

have been obtained by an hybrid coarsening strategy.

● Parallel implementation (C++) of EGMRES method and integration in

a reservoir simulator

● Experiment EGMRES for pressure solve (with BJ precond) in a

reservoir simulator