A Fortran 95 object-oriented parallel code for the ...Introduction NEMO main features Examples A...
Transcript of A Fortran 95 object-oriented parallel code for the ...Introduction NEMO main features Examples A...
IntroductionNEMO main features
Examples
A Fortran 95 object-orientedparallel code for the
numerical solution of PDEsNEMO (Numerical Engine for Multiphysics Operators)
Marco Rorro
e-mail:[email protected] Inter-Univ. Computing Consortium
Seminario di Modellistica differenziale numericaRome, October 27, 2009
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Outline
1 IntroductionObject-Oriented programming in Fortran 95The PSBLAS library
2 NEMO main featuresPartitioning and ReorderingGeometry and mesh capabilitiesPDE solver
3 ExamplesHeat transferConvection-diffusionIncompressible Navier-Stokes equations
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Main references
S. Toninel, Development of a New Parallel Code for ComputationalContinuum Mechanics Using Object-Oriented Techniques, PhD Thesis,Universitá di Bologna, 2006.(http://www.ce.uniroma2.it/nemo/toninel_phd.pdf)
J.H. Ferziger and M. Peric, Computational Methods for Fluid Dynamics,Third ed. Springer, 2002.
S. Filippone and A. Buttari, PSBLAS-2.0 User’s Guide. A ReferenceGuide for Parallel Sparse BLAS library, 2006.
S. V. Patankar, Numerical Heat Transfer and Fluid Flow, First ed. Seriesin Computational Methods in Mechanics and Thermal Sciences. Taylor& Francis, 1980.
H. K. Versteeg and W. Malalasekera, An Introduction to ComputationalFluid Dynamics - The Finite Volume Method, First ed. Prentice Hall,1995.
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Main references
S. Toninel, Development of a New Parallel Code for ComputationalContinuum Mechanics Using Object-Oriented Techniques, PhD Thesis,Universitá di Bologna, 2006.(http://www.ce.uniroma2.it/nemo/toninel_phd.pdf)
J.H. Ferziger and M. Peric, Computational Methods for Fluid Dynamics,Third ed. Springer, 2002.
S. Filippone and A. Buttari, PSBLAS-2.0 User’s Guide. A ReferenceGuide for Parallel Sparse BLAS library, 2006.
S. V. Patankar, Numerical Heat Transfer and Fluid Flow, First ed. Seriesin Computational Methods in Mechanics and Thermal Sciences. Taylor& Francis, 1980.
H. K. Versteeg and W. Malalasekera, An Introduction to ComputationalFluid Dynamics - The Finite Volume Method, First ed. Prentice Hall,1995.
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Object-Oriented programming in Fortran 95The PSBLAS library
OO languages for scientific computing
The programming language chosen for supporting anobject-oriented formulation should satisfy the followingrequisites:
Data abstractionEncapsulationInheritancePolymorphismOperator overloading
Besides this, one should also account forcomputational efficiencydebugging and maintaining possibilitiescross-platform portability
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Object-Oriented programming in Fortran 95The PSBLAS library
NEMO and PSBLAS
NEMO is designed for running on distributed memoryparallel systems, according to the Message Passingparadigm. In particular the numerical algorithms whichrequire great effort in order to be parallelized are theiterative methods for the solution of the linear systemarising from the discretization of the PDE.This task is accomplished by means of the Parallel SparseBLAS library (http://www.ce.uniroma2.it/psblas)whose development was addressed by S. Filippone and M.Colajanni (2000) and S. Filippone and A. Buttari.
It provides a large set of very efficient preconditioners andnon-stationary, iterative, methods from the Krylov subspacefamily.Preconditioners and iterative methods are invoked bycalling single subroutines in a manner completelytransparent to the user.
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Object-Oriented programming in Fortran 95The PSBLAS library
NEMO and PSBLAS
NEMO is designed for running on distributed memoryparallel systems, according to the Message Passingparadigm. In particular the numerical algorithms whichrequire great effort in order to be parallelized are theiterative methods for the solution of the linear systemarising from the discretization of the PDE.This task is accomplished by means of the Parallel SparseBLAS library (http://www.ce.uniroma2.it/psblas)whose development was addressed by S. Filippone and M.Colajanni (2000) and S. Filippone and A. Buttari.
It provides a large set of very efficient preconditioners andnon-stationary, iterative, methods from the Krylov subspacefamily.Preconditioners and iterative methods are invoked bycalling single subroutines in a manner completelytransparent to the user.
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Object-Oriented programming in Fortran 95The PSBLAS library
The PSBLAS library consist of
computational routinessparse matrix by dense matrix productsparse triangular system solution for block diagonalmatricesvector and matrix normdense matrix sumsdot product
auxiliary routinescommunication descriptor allocationdense and sparse matrix allocationdense and sparse matrix build and updatesparse matrix and data distribution preprocessing
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Object-Oriented programming in Fortran 95The PSBLAS library
The PSBLAS library consist of
Preconditioner routinesDiagonal scalingIncomplete LU (ILU) factorizationAdditive SchwarzMultilevel Schwarz
Iterative methodsConjugate Gradient (CG)Generalized minimal Residual (GMRES)BiConjugate Gradient (BICG)Conjugate Gradient Stabilized (CGS)BiConjugate Gradient Stabilized (BiCGSTAB)
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Partitioning
case(-1)call bld_part_random(ncells,nprocs,part_cells)
case(0)call bld_part_block(ncells,nprocs,part_cells)
case(1)call get_conn_csr(c2c,xadj_glob,adjncy_glob)call bld_part_graph(xadj_glob,adjncy_glob,ipart=1)call get_part_graph(part_cells)
case defaultwrite(*,200)call abort_psblas
end select
call psb_cdall(icontxt,desc_c,info,vg=part_cells)
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Partitioning example
Block partition Metis partitition
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Reordering
In order to reduce the bandwith and profile of connectivitymatrix A
Bandwith = max{|i − j | : aij 6= 0
}(1)
Profile =N∑
i=1
di , di = i − fi , fi = min{j : aij 6= 0} (2)
NEMO implement the reordering algorithm developed by Gibbset al. (Gibbs, Poole and Stockmeyer An algorithm for reducingthe bandwith and the profile of a sparse matrix, SIAM J. ofNum. Anal., 1976).The renumbering improve also the partition.
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Reordering example
sparsity pattern without reordering sparsity pattern with GPS reordering
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Reordering + partitioning example
Block partition without reordering Block partitition with GPS reordering
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Mesh main features I
triangles, quadrilaterals or generic polygons (2D)tetrahedra, pyramids, prism, hexahedra or genericpolyhedra (3D)
3D hybrid mesh with mixed tetrahedra, pyramids, prism and hexaedra
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Mesh main features II
co-located arrangmentmaster-slave, face-based approach
Master-slave approach
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Supported mesh formats
Input formats:Gambit Neutral Format (grid generator of FLUENT)CGNS format (CFD General Notation System)(http://cgns.sourceforge.net/)
Output formatDX (http://www.opendx.org/)VTK Visualization Toolkit (http://www.vtk.org/)In this case the results can be visualized for example with
VISIT (https://wci.llnl.gov/codes/visit/)ParaView (http://www.paraview.org/)
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Vector, Face and Cell class
UML diagram of Vector, Face and Cell classes
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
The vector class I
module class_vectorprivate ! Defaultpublic :: vector ! Classpublic :: vector_, alloc_vector ! Constructorpublic :: free_vector ! Destructorpublic :: x_, y_, z_ ! Getterspublic :: operator(+), operator(-), &
operator(*) , ... ! Vector Algebra
type vectorprivatereal(kind(1.d0)) :: xreal(kind(1.d0)) :: yreal(kind(1.d0)) :: z
end type vectorMarco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
The vector class II
interface operator(+)module procedure vec_sum
end interface...containselemental function vec_sum(a,b)type(vector) :: vec_sumtype(vector), intent(in) :: a, bvec_sum%x = a%x + b%xvec_sum%y = a%y + b%yvec_sum%z = a%z + b%z
end function vec_sum...end module class_vector
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
The mesh class
UML diagram of the mesh class
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
The field class
UML diagram of the field class
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
The PDE class
UML diagram of the PDE class
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
Operators
Operators acting on Field ObjectsGradient of scalar and vector fieldsDivergence of vector field
Operators acting on PDEsTime derivative of scalar and vector variablesLaplacian (div(Γgradφ)) of scalar and vector variablesDivergence of scalar variables and vector variablesScalar Source
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
subroutine scalar_pde_div(sign,pde,phi,phi2,flux,...)
...! Gets local to global conversion listcall psb_get_loc_to_glob(msh%desc_c,iloc_to_glob)
...call get_ith_conn(if2b,msh%f2b,0)insert_fluid: do
...! Local indicesi = i + 1if = if2b(i)im = master_(msh%faces(if))is = slave_(msh%faces(if))! Global indicesim_glob = iloc_to_glob(im)is_glob = iloc_to_glob(is)
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver
subroutine scalar_pde_div(sign,pde,phi,phi2,flux,...)
coeff_a(1) = fact * (1-w); coeff_a(2) = fact * w! Diagonal coefficients ! MasterA(ka) = coeff_a(1)irow_a(ka) = im_glob; icol_a(ka) = im_glob! SlaveA(ka+1) = coeff_a(2)irow_a(ka+1) = im_glob; icol_a(ka+1) = is_glob! Off-diagonal coefficients ! Master rowA(ka+2) = -coeff_a(1);irow_a(ka+2) = is_glob; icol_a(ka+2) = is_glob! Slave rowA(ka+3) = -coeff_a(2)irow_a(ka+3) = is_glob; icol_a(ka+3) = im_glob
end do block_fluid_cdcall spins_pde(nel,irow_a,icol_a,A,pde)...
end do insert_fluidMarco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
div(λgradT) + ST = 0
- Number of cells: 6813804- Number of faces: 13696956- Number of vertices: 1169649
BC1 and BC2: adiabatic wall (∂T/∂n = 0) BC3: wall (T = 300)
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Heat transfer program (variables declaration)
program steady_conductionuse tools_nemocharacter(len=30), parameter :: input_file = ’nemo.inp’type(mesh) :: mshtype(material) :: mattype(bc_poly), allocatable :: bc(:)type(scalar_field) :: T, lambdatype(scalar_pde) :: eqnTtype(scalar_source) :: srctype(iterating) :: itertype(output) :: outreal(kind(1.d0)) :: var
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Heat transfer program (variables setup)
call start_psblas
call create_mesh(msh,input_file,’MESH’)call create_material(mat,input_file,’MATERIAL’)call create_bc(bc,input_file,’BC’,msh)
call create_field(T,msh,bc=bc,mat=mat,dim=temperature_)call create_field(lambda,msh,bc=bc,mat=mat,&
dim=conductivity_,on_faces=.true.)
call create_pde(eqnT,input_file,’Energy PDE’,msh,power_)call create_source(src,input_file,’SOURCE’,power_ / volume_)call create_iterating(iter,input_file,&
’BIG ITERATIONS’,it_convergence_)
call create_output(out,input_file,’OUTPUT’)
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Heat transfer program (PDE solving)
big_iteration: docall increment(iter)if(mypnum_() == 0) write(*,100) current_iteration(iter)
call pde_laplacian(’+’,eqnT,lambda,T)call pde_source(’+’,eqnT,src)
call solve_pde(eqnT,T,var=var)
call update_field(lambda,T)
if(stop_iterating(iter,var)) exit big_iteration
end do big_iteration
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Heat transfer
Execution time: 2728 s. 1 core, 810 8 cores on one node of thematrix cluster at CASPUR
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Heat transfer
BC1: fixed flux (∂T/∂n = −1.6d4), BC2: fixed temp (T = 510), BC3: fixed temp (T = 360), BC4 and BC5:adiabatic (∂T/∂n = 0), BC6: fixed flux (∂T/∂n = −0.6d4), BC7: fixed temp (T = 540).
Nubmer of cells: 416469, Number of faces: 857563, Number of vertices: 81738
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Heat transfer
Execution time: 76 s. on my laptopMarco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Convection-Diffusion
div(vφ) + div(Γgradφ) = 0
Geometry and boundary conditions for the scalar transport in a stagnation point flow
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Convection-Diffusion
Γ = 0.01 Γ = 0.001
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Navier-Stokes equations for an incompressible flow
div(v) = 0 continuity eq.∂v∂t
+ div(vv) = div(νgradv)− 1ρ
gradp momentum eq.
The second equation in Cartesian form reads:
∂ui
∂t+ div(uiv) = div(νgradui)−
1ρ
divpii
where ui (i = 1,2,3) are the Cartesian component of thevelocity vector v and ν = µ/ρ is the kinematic viscosity(dynamic viscosity / density).
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Implicit pressure-correction methods
Take the divergence of the momentum equation:
div(
1ρ
gradp)
= −div[∂v∂t
+ div(vv − νgradv)
]exploit the continuity equation to get:
div(
1ρ
gradp)
= −div [div(vv)]
In Cartesian coordinates this equation reads:
∂
∂xi
(1ρ
∂p∂xi
)= − ∂
∂xi
(∂
∂xj(uiuj)
)
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Semi-Implicit Method for Pressure-Linked Equations
Let um∗ the solution of the momentum eq. with p = pm−1:
AuiP um∗
i +∑
n
Auin um∗
i,n = Bm−1ui−(
1ρ
∂pm−1
∂xi
)P
Correction: umi = um∗
i + u′ and pm = pm−1 + p′
substitute this expressions in the momentum equation to get:
u′i,P =1
AuiP
[1ρ
(∂p′
∂xi
)P−∑
n
Anu′i,n
]apply the continuity eq. to um, by neglecting divu′ get:
∂
∂xi
(1
AuiP
1ρ
∂p′
∂xi
)=∂um∗
i∂xi
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Lid-driven cavity
pressure for Re = vL ∗ L/ν = 1000 velocity magnitude for Re = vL ∗ L/ν = 1000
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Heat transferConvection-diffusionIncompressible Navier-Stokes equations
Lid-driven cavity movie
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators
IntroductionNEMO main features
Examples
Future work
Test the SIMPLE schemeImplement other pressure correction schemes, like PISOAdd turbolent models
Marco Rorro NEMO: Numerical Engine for Multiphysics Operators