Parallel Left Ventricle Simulation Using the

FEniCS Framework

Timofei Epanchintsev and Vladimir Zverev

eti@imm.uran.ru

2nd Ural Workshop on Parallel, Distributed, and Cloud Computing for Young Scientists

Anatomy of the Heart

2

L. Sherwood, Human physiology : from cells to systems. Australia ; United Kingdom:

Thomson Brooks/Cole, 2004.

Heart Simulation

3

Tissue OrganCell

• Multiscale modeling• Cell

• Tissue

• Organ

• Heart simulation requires a lot of time• Parallel computing is needed

Heart Simulation Obstacles

• Difficult code development for HPC environment and its maintenance• porting complicated code requires long time (3-

5 years). Modern architecture can become obsolete;

• adaptation for HPC architecture often leads to significant changes of code. Difficult to change the mathematical models and numerical methods in the optimized program

4

Automated Scientific Computing Frameworks

5

• Automated Scientific Computing Frameworks• OpenFOAM• OpenCMISS• Chaste• FEniCS

• Advantages• Software development using high level tools• Automatic parallelization

• Disadvantage• Low performance

• Performance improvements• Just-in-time compilers, high performance mathematical libraries, etc.

The FEniCS framework

6

7

• Ekaterinburg-Oxford [1] model describes electrical, chemical, and mechanical processes

𝑑𝑆

𝑑𝑡= 𝑔(𝑉, 𝑆)

• 𝑉 is the electrical potential, 𝑆 is the vector of state variables (the dimension is 30) that governs the ion currents

𝑑𝑉

𝑑𝑡= 𝐷𝛻2𝑉 + 𝐼𝑖𝑜𝑛𝑠(V, S)

• 𝐷 is the diffusion matrix, 𝛻2 is the Laplace operator, and 𝐼𝑖𝑜𝑛𝑠 is the sum of the ionic currents

1. Solovyova, O., Vikulova, N., Katsnelson, L.B., Markhasin, V.S., Noble, P., Garny, A., Kohl, P.,

Noble, D.: Mechanical interaction of heterogeneous cardiac muscle segments in silico: effects on Ca 2+ handling and action potential. International Journal of Bifurcation and Chaos 13(12) (2003)

Model of the Heart Electrical Activity

Electric Potential and Ionic Currents

8

Benchmark Problem

3D asymmetric left ventricle model [1]

• Parameters were captured by ultrasound

Implementation• Operator splitting scheme

of the first order [2];• Implicit method usage;• No Newton-like iterations.

9

1. Pravdin, S.F., Berdyshev, V.I., Panfilov, A.V., Katsnelson, L.B., Solovyova, O., Markhasin, V.S.: Mathematical model of the anatomy and fibre orientation field of the left ventricle of the heart. Biomedical engineering online 54(12) (2013)2. Li, Y., Chen, C.: An efficient split-operator scheme for 2-D advection-diffusion simulations using finite elements and characteristics. Applied Mathematical Modelling 13(4) (1989) 248–253

FEniCS Program Example# Formulation of the diffusion PDE variational problemmesh = Mesh() # Code for loading mesh from the file

# Building function space for action potentialSpace_AP = FunctionSpace(mesh, "Lagrange", lagrange_order)

# Define the PDE Problemv = TrialFunction(Space_AP)v0 = Function(Space_AP)PdePart = (1.0/dt)*inner(v - v0, q1)*dx \

- (-inner(D*grad(v), grad(q1)))*dxPDEproblem = LinearVariationalProblem(lhs(PdePart),

rhs(PdePart), v, bcs=bcs)

# Creating the PDE solverPDEsolver = LinearVariationalSolver(PDEproblem)

10

FEniCS Program Example

# Solving the differential equation systemsfor t in time_range[1:]:

# Solving diffusion equation assign(v0, v)PDEsolver.solve()

# Solving cell equationsassign(ode_vars0, ode_vars)assign(ode_vars0.sub(0), v)ODEsolver.solve()

# Storing data if necessaryif steps % saving_step_interval == 0:

v_file << (v, t)steps += 1

11

Krylov Solvers and Preconditioners in FEniCS

12

Solver Preconditioner

Biconjugate gradient stabilized method Algebraic multigrid

Conjugate gradient method Default preconditioner (bjacobi + ilu)

Generalized minimal residual method Hypre parallel incomplete LU factorization

Minimal residual method Hypre parallel sparse approximate inverse

Richardson method Incomplete Cholesky factorization

Transpose-free quasi-minimal residual method

Incomplete LU factorization

Hypre algebraic multigrid

Successive over-relaxation

Computational Experiment

• Supercomputer “URAN”• CPU - 2 x Intel(R) Xeon(R) CPU X5675 @ 3.07GHz

• RAM - 192 GB

• Interconnect - Infiniband DDR (20 Gbit)

• OS - Red Hat Enterprise Linux 6.7

• FEniCS version 1.6.0

13

Computational Experiment

• Mesh generated by GMSH• size of tetrahedron is 2-4mm;

• mesh contains 7178 points and 26156 tetrahedrons.

• Initial conditions• activation of an entire LV (action potential = 40 mV)

• Time step 2.5e-5 seconds• in order to capture fast processes.

• The simulation period 0.3 seconds• electrical activity tends to the equilibrium state.

14

Performance of Krylov Solvers and Preconditioners

15

Solver Preconditioner Simulation Time, sec

Minimal residual method

Successive over-relaxation 82.78

Conjugate gradient method

Successive over-relaxation 83.03

Generalized minimal residual method

Hypre algebraic multigrid 988.94

Generalized minimal residual method

Default preconditioner (bjacobi + ilu)

985.24

The Simulation Time

16

Minimal residual method solver, Successive over-relaxation preconditioner

Simulation Speedup

17

Minimal residual method solver, Successive over-relaxation preconditioner

Comparison with manual implementation

LeVen• Our previous implementation [1]• C language• OpenMP for shared memory

systems• Limited scalability

Comparison with LeVen• Same performance• Better scalability

18

1. Sozykin, A., Pravdin, S., Koshelev, A., Zverev, V., Ushenin, K., Solovyova, O.: LeVen - a parallel system for simulation of the heart left ventricle. 9th IEEE International Conference on Application of Information and Communication Technologies, AICT 2015 Proceedings (2015) 249–252

Conclusion

FEniCS framework is an efficient tool for heart simulation on parallel computing systems

• Near-mathematical notation

• Automatic parallelization on MPI clusters

Performance testing case

• Simulation of heart electrical activity using Ekaterinburg-Oxford model

• Approximately the same performance as the implementation using the C programming language

• 90x speedup using the 240 CPU cores

• Near-linear scalability

• The type of Krylov linear solver and preconditioner significantly influence the simulation time

19

Thank you for your attention!

Epanchintsev Timofei

• eti@imm.uran.ru

Parallel Left Ventricle Simulation Using theFEniCS Framework

This work was supported by the Russian Science Foundation (grant no. 14-35-00005). Our study was performed using the Uransupercomputer of the Krasovskii Institute of Mathematics and Mechanics.

20

21