An Efficient Multi-Threaded Newton-Raphson Algorithm for StrongCoupling Modeling of Multi-Physics ProblemsAmir Akbari∗, Dennis GiannacopoulosMcGill University, Department of Electrical and Computer Engineering, Montreal, Canada

ART ICLE INFOKeywords:Multi-physics problemsNewton-Raphson methodGaussian belief propagationParallel scalabilityRadiofrequency ablation

ABSTRACTThe Finite Element Gaussian Belief Propagation (FGaBP) method is an iterative algorithm with abun-dant parallelism making it an alternative for the traditional Finite Element Method (FEM), especiallyfor large multi-physics problems. In this paper, we extend the FGaBP method to solve the coupledelectrical-thermal problem that emerges in the modeling of radiofrequency ablation (RFA) of hepatictumors. The strongest form of coupling algorithms, which is the Newton-Raphson (NR) method, isimplemented in parallel using the localized computations of FGaBP. The parallel scalability of theFGaBP method is retained in the proposed algorithm by calculating local Jacobian matrices for eachelement and then updating the solutions for both electrical and thermal problems accordingly at eachNR iteration.

1. IntroductionIn general, finite element method (FEM) implementa-

tions consist of two computationally expensive stages whichare the sparse matrix assembly stage, and the solving stageof the linear system using iterative solvers. These two stagesare even more expensive for non-linear applications. Whilefor certain cases, the assembly needs to be done only onceand, therefore, its cost can be tolerated, for non-linear prob-lems, such as multi-physics applications, the assembly stagecan dominate the solving stage. This is especially so for apure Newton-Raphson (NR) method, when the constructionof a Jacobian matrix at each linearizing iteration could beprohibitively expensive for large scale problems.

The Jacobian-free Newton-Krylov (JFNK) method is themost widely used inexact solver for the NR algorithm. Thismethod employs a Krylov-based iterative solver in which theJacobian matrix does not have to be explicitly formed, and inthis way, JFNK bypasses the main obstacle associated withthe assembly stage of the NR method. On the other hand,in the solving stage, the efficiency of JFNK depends criti-cally on preconditioning the inner Krylov subspace method.It is in this area that the Jacobian-free appeal of JFNK mustyield to the construction and use of a preconditioning matrixwhich require the execution of a number of global algebraicoperations in each JFNK iteration, such as a Sparse Matrix-Vector Multiplication (SMVM). The SMVM operation, inparticular, can strongly limit the acceleration of the solvingstage using parallel processing due to its dependency on theunderlying sparse data-structure [10]. In addition, the ap-proximation error associated with the Jacobian-vector mul-tiplication represents the greatest disadvantage of JFNK, es-pecially when variables associated with different physics be-ing coupled in a multi-physics application differ by orders ofmagnitude.

∗Corresponding authoramir.akbari3@mail.mcgill.ca (A. Akbari);

dennis.giannacopoulos@mcgill.ca (D. Giannacopoulos)ORCID(s):

The Gaussian belief propagation (GaBP) method is a re-cursivemessage passing algorithm that offers distributed com-putation providing potential to parallelize FEM applications[11]. This work presents a NR reformulation of GaBP in or-der to exploit its localized computations and message pass-ing scheme for solving multi-physics problems in parallel.Similar to the JFNK, GaBP does not need to explicitly form aglobal Jacobian matrix, instead, local computations are per-formed to calculate local Jacobian matrices for each elementin parallel. This provides a NR algorithm amicable to differ-ent parallel computing architectures. On the other hand, incontrast to the JFNK, there is no approximation associatedwith the local Jacobian matrices which makes the novel NRmethod more accurate than the JFNK method.

As a multi-physics application, the new method is ap-plied to the coupled electrical-thermal phenomenon that ap-pears in radiofrequency ablation (RFA) of liver tumors. RFAdelivers electrical currents to biological tissues to thermallydamage a tumor by raising its temperature for a certain dura-tion. The purpose of an effective RFA is the entire ablationof the tumor with a safety margin of destroyed healthy tis-sue in its vicinity, which in turn relies on the extent of theablation zone. However for treating tumors in the liver, thehepatic blood vessels dissipate heat and change the size andshape of the lesion zone [3]. Recently, computational mod-els of RFA have gained attention as a tool for studying theheat distribution inside the tissue [15, 27]. To investigatethe capabilities of our NR method, the coupled electrical-thermal phenomenon in a simplified model of RFA is simu-lated in this paper.

The paper is organized as follows. In Section 2, a sum-mary of the coupled phenomena in RFA is provided. InSection 3, we present the GaBP formulation for the coupledelectrical-thermal problem in RFA, and a parallel NR for-mulation based on the GaBP representation of the problem.Finally, in Sections 5 and 6, we present the results and con-cluding remarks, respectively.

: Preprint submitted to Elsevier Page 1 of 10

Please cite as: Amir Akbari, Dennis Giannacopoulos, An efficient multi-threaded Newton–Raphson algorithm for strong coupling modeling of multi-physics problems, Computer Physics Communications, Volume 258, 2021, 107563, ISSN 0010-4655,https://doi.org/10.1016/j.cpc.2020.107563.(https://www.sciencedirect.com/science/article/pii/S0010465520302708)

2. The multi-physics phenomena in RFARFA probes operate between 460– 550 kHz. At these

frequencies, the wavelength is several orders of magnitudelarger than the size of the ablation electrode. Thus, the dis-placement current is neglected and a quasi-static electricalconduction model is assumed, which allows us to solve theelectric potential around the probe by using Laplace’s equa-tion:∇ ⋅ (�(T )∇v) = 0, (1)where �(T ) is the temperature-dependent electrical conduc-tivity (S/m), and v is the electric potential (V). In the lit-erature, �(T ) is usually introduced by mathematical func-tions which reflect its dependence on temperature. Accord-ing to [28], the most commonly used mathematical functionfor �(T ) is a piecewise function which uses different mathe-matical expressions according to the temperature range. Attemperatures below 100 oC, �(T ) increases with temperaturewith a rate from 1.5%∕oC to 2%∕oC, either linearly or expo-nentially [28]. At temperatures above 100 oC, a rapid dropof �(T ) occurs due to vaporisation. However, the computersimulation of RFA could be ended when the maximum tem-perature in the tissue reaches 100 oC [8], considering the factthat the RF clinical power delivery protocol is usually basedon varying the voltage applied between electrodes to keepthe temperature at the electrode tip set to 80–90 oC [20].

Solving Laplace’s equation has a high computational costsince it requires a fine volumemesh on the surface of the verythin probe tips. Thus, Audigier et al [3] model the electricalheating with a Dirichlet boundary condition on a sphere witha pre-determined radius around the probe tip. Another ap-proach to approximate the Joule heating is proposed by [18]and used in later RFA simulation frameworks [17], [29] and[23]. In this approach, the power deposition is estimated us-ing a Gaussian distribution around the probe tip. However,these approximations are, potentially, too much of a simpli-fication as it has been shown by [14] that electrical parame-ters are critical in models of hepatic RFA. In addition, noneof these approaches consider the temperature dependency of� and the coupling between the electrical problem and thethermal problem. The most commonly used model for heattransfer in the tissue is the Pennes model [19]:

�ticti)T)t

= Q + ∇ ⋅ (d∇T ) +H(Tbl − T ), (2)

where T is the temperature (K), cti is the special heat ca-pacity of tissue (J/kg/K), �ti is the tissue density (kg∕m3),d is the thermal conductivity (W/m/K),H is the convectivetransfer coefficient (W∕m3∕K) and Tbl is the baseline phys-iological blood temperature taken to be 310 K. On the righthand side of (2), Q = �|∇v|2 is the heat source (W∕m3)which depends on the electric potential. On the other hand,in the electrical problem � is changing with the tempera-ture. As a result, the electrical and thermal problems arecoupled to each other and amulti-physics approach is neededto model this phenomenon.

In the Pennes model, the blood temperature is assumedconstant, which is only valid within and close to large ves-sels [3]. For this reason, Payne et al [18] proposes a multi-physicsmodel for computing heat diffusion in the liver, whereeach elementary volume is assumed to comprise both tissueand blood with a certain fraction:(1 − �)�ticti

)Tti)t

= (1 − �)Q + (1 − �)∇ ⋅ (d∇Tti) (3a)+H(Tbl − Tti),

��blcbl()Tbl)t

+ v.∇Tbl) = �Q + �∇ ⋅ (d∇Tbl) (3b)−H(Tbl − Tti).

In theses two coupled equations, � stands for blood volumefraction (fraction of blood volume over total volume), v isthe blood velocity (m/s) , and subscripts ‘ti’ and ‘bl’ refer totissue and blood phase respectively. Most RFA theoreticalmodels have used a constant value for d, probably due to thefact that changes in d with temperature are not so marked asin � [25]. The numerical values of the electrical and thermalmodel parameters used in the literature are provided in Ta-ble 1. Because if its high computational cost, the electricalproblem and its coupling to the thermal problem has oftenbeen over-simplified in the recent literature. This paper triesto develop an efficient multi-physics solver and applies it tothe coupled electrical-thermal problem in RFA.

Table 1Values of electrical and thermal model parameters used in theliterature.

Parameter Value Ref

Baseline liver electrical conductivity, �ti (S/m) 0.33 [9]Probe electrical conductivity, �probe (S/m) 108 [9]Convective transfer coefficient, H (W(m3 K)−1) 24.4×105 [3]Liver density, �ti (kg/m3) 1060 [18]Liver heat capacity, cti (J/kg⋅ K) 3600 [18]Baseline liver thermal conductivity, dti (W/m⋅ K) 0.512 [18]Blood density, �bl (kg/m3) 1000 [9]Blood heat capacity, cbl (J/kg⋅ K) 4180 [18]Probe density, �probe (kg/m3) 6450 [9]Probe heat capacity, cprobe (J/kg⋅ K) 840 [9]Probe thermal conductivity, dprobe (W/m⋅ K) 18 [9]

3. Parallel formulation of the coupledelectrical-thermal problemAs explained in Section 2, the focus of this paper is on

the coupled electrical-thermal problem. For this reason, thecoupling between the tissue temperature and the blood tem-perature, i.e., (3) is neglected and the thermal problem ismodeled by (2). If the theta-scheme [26] is used for timediscretization of (2), and similar FEM mesh for both (1) and(2), we will have the following set of coupled discrete equa-tions:[

Mv(T )] {

v}

= 0 (4a): Preprint submitted to Elsevier Page 2 of 10

Figure 1: (Left): Sample FEM mesh of two second order tri-angles. (Right): Two types of nodes in the factor graph rep-resentation.

[

MT + �t � ST] {

T}n+1 =

[

MT − �t (1 − �)ST] {

T}n

(4b)+ �t

{

(1 − �){

f(v)}n + �

{

f(v)}n+1}

where Mv,ij(T ) = ∫ �(T )⃖⃖⃗∇Ni ⋅ ⃖⃖⃗∇Nj dV depends on thetemperature, fj(v) = ∫ (HTbl + �(T )|∇v|2)Nj dV dependson the potential, MT ,ij = (�cti + �tH) ∫ NiNj dV andST ,ij = d ∫ ⃖⃖⃗∇Ni ⋅ ⃖⃖⃗∇Nj dV . Note that Ni and Nj are FEMscalar basis functions, n is the time step number, �t is thetime step value and 0 ≤ � ≤ 1 is the parameter of the theta-scheme time discretization. Because of straightforward im-plementation, successive substitution is often the first routeto a multi-physics simulation. This approach, which is con-sidered a weak-coupling model iterates over the uni-physicsproblems, solving the first equation for the first unknown,given the second unknown, and the second equation for thesecond unknown, given the first. The main advantage ofsuccessive substitution is making use of the existing codesfor the uni-physics problems; however, it may ignore strongcouplings between physics and give a false sense of com-pletion [16]. If a solver code that goes beyond the weak-coupling is needed, the NRmethod is the simplest algorithm.This section provides an efficient FEM-based NR algorithmfor solving the coupled electrical-thermal problem of (4) ina strong coupling manner. For this purpose, a parallel FEMformulation based on GaBP algorithm introduced by [10]is modified for solving multi-physics problems. The FEM-based GaBP (FGaBP) method was tested on simple electro-static problems showing good parallel scalability comparedto other parallel algorithms [10]. This paper extends FGaBPto deal with a multi-physics scheme.

The main idea of FGaBP is to reformulate FEM into aninference problem over a factor graph, which may be solvedapplying the Gaussian belief propagation rules. As shownin Fig. 1, FGaBP turns the FEM mesh into a factor graph.Two different types of nodes are then introduced in the factorgraph: variable nodes (VN, nodes of unknowns) representedby circles, and factor nodes (FNs) represented by squares.The solution at each VN is considered a random variablewith a Gaussian distribution whose shape is defined by twoparameters, � and �, where � is the reciprocal of the varianceand �∕� is the mean. By passing messages between each FNand all its connected VNs, the GaBP algorithm tries to findthe values of � and � for each VN. A message, mai, is sentfrom factor node a (FNa) to the connected variable node i(VNi) and represents the most probable solution value at i,

as observed from FNa. In return, VNi sends a message backto FNa representing observations from other connected FNs.The following is the formulation of the FGaBP algorithmupdate rules[10]:

1. t = 0: Initialize all messages �(0) = 0 and �(0) = 1.2. Iterate: t = 1, 2, ... and t∗ ≤ t.

(a) For each VNi, compute messages �ia and �ia toeach connected FNa (a ∈ (i)) as follows:�(t∗)i =

∑

k∈ (i)�(t∗)ki , �(t)ia = �

(t∗)i − �(t∗)ai (5)

�(t∗)i =∑

k∈ (i)�(t∗)ki , �(t)ia = �

(t∗)i − �(t∗)ai (6)

where (i) is the neighborhood set of node i.(b) For each FNa:

i. receive messages �(t)ia and �(t)ia where i ∈ (a).

ii. Assume(t∗) is a diagonal matrix of incom-ing �(t∗)ia messages, and (t∗) is a vector ofincoming �(t∗)ia messages, then define matrixW and vector K as follows:W(t∗) =M +(t∗) (7)

K(t∗) = B + (t∗) (8)whereM and B are element a characteristicmatrix and source vector respectively.

iii. Partition W(t∗) and K(t∗) as follows:

W(t∗) =

[

W (t∗)(i) VT

V W̄(t∗)

]

(9)

K(t∗) =

[

K (t∗)(i)

K̄(t∗)

]

(10)

where (i) is the local index correspondingto the global variable node i.

iv. Compute and partition (W (t∗))−1 as follows:

(W(t∗))−1 =[

W̃(i) C̃T

C̃ W̃

]

. (11)

v. Compute and send new FNa messages �(t)aiand �(t)ai to each VNi as follows:

�(t)ai =1

W̃(i)− �(t∗)ia . (12)

�(t)ai = B(i) +1

W̃(i)(K̄(t∗))T C̃T . (13)

: Preprint submitted to Elsevier Page 3 of 10

Figure 2: (Left): A message mai is sent from FNa to VNi.(Right): A message mia is sent back from VNi to FNa.

Figure 3: (Left): A message mai,v (mai,T ) is sent from FNa toVNi in the electrical (thermal) problem . (Right): A messagemia,v (mia,T ) is sent back from VNi to FNa in the electrical(thermal) problem.

3. At message convergence, the mean of the VNs, or so-lutions, can be obtained by:

u(t)i =�(t)i�(t)i

(14)

where�i =

∑

k∈N(i)�ki, �i =

∑

k∈N(i)�ki . (15)

The message passing scheme of FGaBP is shown graphi-cally by Fig. 2. In a uni-physics problem, each messagecontains information about the value of the unknown param-eter at each node. For example, in Fig. 2, left, the messagemai in the electrical problem carries information on the po-tential at node i, vi. On the other hand, the different physicsinvolved in a multi-physics problem need to exchange in-formation with each other; if a strong coupling approach isneeded, the exchanged messages between different physicsmust contain information about their sensitivity with respectto each other, besides information about their values. Figure3 shows the message passing scheme in a multi-physics sce-nario. Here, a massage mai,v (mai,T ) sent in the electrical(thermal) problem carries information about vi (Ti) and alsodvi∕dTj (dTi∕dvj).When the update rules of (12) and (13) are applied to thecoupled electrical-thermal problem, it should be noted that

according to (4), in the electrical problem, element a char-acteristic matrix, Mv, depends on the temperature values atnodes i when i ∈ (a). On the other hand, in the thermalproblem, element a source vector, BT , depends on the elec-tric potential values at nodes i where i ∈ (a). As a result,equations (11) and (12) can be rewritten as:

�(t)ai,v =1

W̃(i),v(Tj)− �(t∗)ia,v , (16a)

�(t)ai,v = B(i),v +1

W̃(i),v(Tj)(K̄v

(t∗)(vj))T C̃v(Tj) , (16b)

�(t)ai,T =1

W̃(i),T− �(t∗)ia,T , (16c)

�(t)ai,T = B(i),T (vj) +1

W̃(i),T(K̄T

(t∗)(vj , Tj))T C̃T , (16d)

where (�ai,v, �ai,v) are messages in the electrical problem,(�ai,T , �ai,T ) are messages in the thermal problem and i, j ∈ (a). Equations (16a) to (16d) present a coupled system ofequations at each FGaBP iteration inside FNa. The rest ofthis section provides a parallel strong coupling algorithm tosolve these coupled equations. According to (16c), � mes-sages in the thermal problem are independent of the poten-tial values; as a result we only consider the coupling between(16a), (16b) and (16d):⎧

⎪

⎪

⎪

⎨

⎪

⎪

⎪

⎩

�(t)ai,v =1

W̃(i),v(Tj )− �(t∗)ia,v ,

�(t)ai,v = B(i),v +1

W̃(i),v(Tj )(K̄v

(t∗)(vj))T C̃v(Tj),

�(t)ai,T = B(i),T (vj) +1

W̃(i),T(K̄T

(t∗)(vj , Tj))T C̃T .

(17)

In general, a multi-dimensional zero-finding problem couldbe written as:F (x) = 0, (18)where F (x) = (f1(x), f2(x), ..., fn(x))T is a system of n cou-pled equations and each fi(x) maps the vector of unknownsx = (x1, x2, ..., xn)T with dimension n into a scalar. TheNR method solves such a non-linear system by solving lin-ear systems successively, that is:J (x(m))Δx(m) = −F (x(m)), (19)where Δx is called the update vector, J (x) is an n-by-n Ja-cobian matrix of F (x), i.e., Jij(x) = )Fi∕)xj , and m is theiteration number. The update vector is then used in order toobtain the solution vector x for the next iteration:x(m+1) = x(m) + Δx(m). (20)

: Preprint submitted to Elsevier Page 4 of 10

Solving a linear system of equations at each iteration canmake NR method prohibitively expensive for a large n. Thissection tries to alleviate this problem by applying the NRmethod at the element-level equations in (17). Suppose k =(i) and l = (j) are the local indices corresponding toglobal VNs i and j, respectively. Since both FGaBP andNR are iterative, we will have two nested iterations at eachtime step. If we are at iteration t (or t∗ ≤ t) of FGaBP and it-eration m of NR, then we will have the following non-linearresiduals for FNa:⎧

⎪

⎪

⎨

⎪

⎪

⎩

�(m,t)ak,v − fk(v(m,t∗)l , T (m−1)l ) = 0

�(m,t)ak,v − gk(v(m,t∗)l , T (m−1)l ) = 0

�(m,t)ak,T − ℎk(v(m−1)l , T (m,t∗)l ) = 0 ,

(21)

where fk, gk and ℎk are the right-hand-sides of the equationsin (17), respectively. At each FGaBP iteration, the sensi-tivities )�ak,v∕)Tl, )�ak,v∕)Tl, )�ak,v∕)vl, )�ak,T ∕)vl, and)�ak,T ∕)Tl are calculated and sent to the neighboring nodesof FNa. After FGaBP iterations have converged, the non-linear system of (17) can be solved with a pure NR method,using local data for each FN. A local Jacobian matrix is con-structed for FNa based on (17) and the sensitivities calcu-lated during FGaBP as follows:Ja =

[

I2nc×2nc Ja,vJa,T Inc×nc

]

3nc×3nc, (22)

where nc is the number of nodes per cell, and I is the iden-tity matrix. The details on the calculation of the off-diagonalelements of Ja are provided in the appendix. The calculationof Ja,v and Ja,T entries needs the partial derivatives of localscalars and vectors with respect to the temperature values atthe nodes in the neighborhood of FNa; As a result, the Ja-cobian matrix associated with FNa, Ja, is only dependent onthe local data structure of FNa, containing a dense matrixMa, vector Ba and messages (�ai, �ai). After FGaBP iter-ations converge and the local Jacobian is formed for FNa,in accordance with (19), �ak,v, �ak,v, and �ak,T updates arecomputed as:⎧

⎪

⎨

⎪

⎩

{

Δ�ak,v}

nc×1{

Δ�ak,v}

nc×1{

Δ�ak,T}

nc×1

⎫

⎪

⎬

⎪

⎭

(m)

=[

J−1a](m)

⋅

⎧

⎪

⎨

⎪

⎩

{

fk − �ak,v}

nc×1{

gk − �ak,v}

nc×1{

ℎk − �ak,T}

nc×1

⎫

⎪

⎬

⎪

⎭

(m)

.

(23)Finally, at the end of the current NR iteration, the messages�ak,v, �ak,v, and �ak,T are updated using the results from (23):⎧

⎪

⎨

⎪

⎩

{

�ak,v}

{

�ak,v}

{

�ak,T}

⎫

⎪

⎬

⎪

⎭

(m+1)

=

⎧

⎪

⎨

⎪

⎩

{

�ak,v}

{

�ak,v}

{

�ak,T}

⎫

⎪

⎬

⎪

⎭

(m)

+

⎧

⎪

⎨

⎪

⎩

{

Δ�ak,v}

{

Δ�ak,v}

{

Δ�ak,T}

⎫

⎪

⎬

⎪

⎭

(m)

. (24)

The updated messages are then used in the next NR itera-tion in the current time step. After the NR iterations haveconverged, we proceed to the next time step. Figure 4 showsthese three nested iterations graphically.

Initialize messages in both problems

Calculate local vectors and matrices

Solve using FGaBP iterations

Calculate local Jacobian matrices

Update the messages

NR IterationsAllsteps

donein

parallel

Repeatforeach

timestep

Figure 4: The parallel NR algorithm.

Figure 5: (Left): The geometry of the test case in two dimen-sions. (Right): Structured quadrilateral mesh containing fourcolors.

4. ImplementationIn this section, the parallel NR algorithm is applied to

the coupled equations in (1) and (2) in both two dimensional(2D) and three dimensional (3D) geometries. As shown inFig. 5, left, tissue surrounding the probe tip is modeled bya 8.0 cm by 8.0 cm square (8.0 cm by 8.0 cm by 8.0 cm cu-bic) domain in 2D (3D), when the 16 V source voltage ismodeled as a Dirichlet boundary condition on the probe lo-cated at the center of the domain. The outer boundaries ofthe domain serve as a return ground electrode. In the ther-mal problem, the normal body temperature, i.e., 37 oC is theinitial value of the temperature, and a Neumann boundarycondition is applied to the outer boundary. The electricaland thermal parameters in (1) and (2) are chosen accordingto Table 1. In addition, the electrical conductivity increaseswith the temperature with a linear rate of 2%∕oC. The algo-rithm is implemented using the open-source FEM softwaredeal.II [7].

An element-based coloring message schedule is imple-mented in order to avoid any race conditions when parallelprocessing is used. The race condition might happen whentwo different elements, e.g., elements a and b in Fig.3, left,try to update the same global node j. If elements a and bbelong to two different threads, then we need to make surethat the messages maj,v and mbj,v do not try to update thevoltage value at node j at the same time. One solution isto schedule the messages based on element coloring. Themesh elements are colored so that no two adjacent elements

: Preprint submitted to Elsevier Page 5 of 10

have the same color symbol. In this way, the messages ineach color group can be computed and safely communicatedin parallel, since elements that belong to the same group donot share any global nodes. A mesh coloring diagram in 2Dis illustrated in Fig. 5, right, using a quadrilateral mesh.1 Start timer;2 Import problem geometry and mesh;3 # pragma omp parallel;4 for NR iteration m = 1, 2,… do5 for each level in the mesh hierarchy do6 # pragma omp for;7 for cell a = 1, 2,… do8 for node i ∈ (a) do9 Initialize messages mai,(v,T );

10 end11 Calculate local vector f and local

matricesMv,T inside cell a, i.e., (4).12 end13 end14 for cycles = 1, 2,… in the V-cycle do15 for Mesh levels from fine to coarse do16 Execute v1 iterations of Algorithm 2;17 Restrict;18 end19 Execute Algorithm 2 on the coarsest level;20 for Mesh levels from coarse to fine do21 Execute v2 iterations of Algorithm 2;22 Prolongate;23 end24 # pragma omp single;25 if global tolerance < tolerance then26 break;27 end28 end29 Execute Algorithm 3 on the finest level;30 # pragma omp single;31 if global residual < NR tolerance then32 break;33 end34 end35 End timer;36 Plot output;Algorithm 1: Parallel NR pseudo-code in each timestep.

Mesh refinement is conduced in 2D (3D) by splittingeach quadrilateral (hexahedral) cell into four (eight) smallercells successively. In this parent-child scheme, we start by acoarse mesh and continue mesh refinement until a fine meshis achieved. In order to accelerate the FGaBP iterations, in-formation from the coarse mesh (Parent) are transferred tothe next fine mesh (Child). The transferred information arethe local messages, mai,v and mai,T calculated for each ele-ment a and node i ∈ (a). In our multi-physics scheme,these messages contain the so called beliefs regarding thetemperature and potential values at each node, as well as the

1 for FGaBP iteration t = 1, 2,… do2 for color c = 1, 2,… do3 # pragma omp for;4 for cell a in color c do5 for node i ∈ (a) do6 Calculate mai(v,T ), i.e., (5) to (13);7 Calculate Ja, i.e., (22) ;8 Update vi and Ti, i.e., (14) ;9 Calculate local message tolerance;

10 end11 end12 # pragma omp critical;13 Update global message tolerance;14 end15 # pragma omp single;16 if global tolerance < FGaBP tolerance then17 return(global tolerance);18 break;19 end20 endAlgorithm 2: FGaBP with local Jacobian calculation

1 for color c = 1, 2,… do2 # pragma omp for;3 for cell a in color c do4 Update mai,(v,T ), i.e., (23), (24);5 Update vi and Ti, i.e., (14);6 Calculate local NR residual;7 Calculate local vector f and local matrices

Mv,T inside cell a, i.e., (4).8 end9 # pragma omp critical;

10 Update global NR residual;11 end

Algorithm 3: NR parallel update

sensitivity of these values with respect to their neighboringnodes. The transfer of information between different refine-ment levels is conducted in a multi-grid scheme. Algorithm1 presents a pseudo-code for the parallel NR method withOpenMP directives. Lines 14 to 28 indicate the multi-gridscheme implemented as a V-Cycle. Algorithms 2 and 3 exe-cute FGaBP iterations and the application of local Jacobianmatrices to update the messages, respectively. Although thesensitivity information is carried on by the messages, onlythe potential and temperature values are considered in theprolongation and restriction; as a result, these two steps areimplemented in parallel exactly as proposed by [12] and arenot described here.

5. ResultsThe numerical results of the newNR formulation are ver-

ified using COMSOL Multiphysics software. The tempera-ture obtained from COMSOL and the newmulti-physics NR

: Preprint submitted to Elsevier Page 6 of 10

Figure 6: Temperature values over time at a specific point,i.e., x = 1 cm and y = 1 cm, when the probe tip is at thecenter of the domain. The time step size �t is equal to 0.5 s.

algorithm at a specific location during oneminute of ablationare shown in Fig. 6. Quantitatively, the Root-Mean-Square(RMS) error between these two simulations is 0.028 oC, whichvalidates our method. To test the parallel scalability proper-ties of themethod, a CPU implementationwithmulti-threading(OpenMP) is provided. As indicated in Algorithm 1, theCPU time calculation includes all the steps except for theoutput of results. All runs are executed on aCompute Canadacluster node. The node contains 2 × 20-core Intel Gold 6148Skylake 2.4 GHz CPUs with 186 GB DRAM. When the pa-rameters v1 = 1 and v2 = 5 in Algorithm 1, the V-Cyclerequired 5 iterations for all 2D and 7 iterations for all 3Druns. This is independent of the number of unknowns inthe finest level which is in agreement with the findings re-ported by [12]. Note that the number of iterations in the in-ner FGaBP execution, i.e., line 19 in Algorithm1, remainsproportional to the problem size in the coarsest level. Inthe electrical problem, we start by a mesh size of 400 cellsin 2D (8000 cells in 3D) which takes 380 (430) inner it-erations to reach a message residual of 10−8. The thermalproblem converges faster in both 2D and and 3D implemen-tations. Figure 7 shows the speedup for a fixed problem sizeof 4,173,281 unknowns in 3D with respect to the numberof processors (strong scaling). A curve is fitted to the plotbased onAmdah’s law [24], fromwhich the ratio of the serialpart of the algorithm (s) is obtained as 0.032. Weak scalingis performed by running the algorithm with different num-bers of threads and with a correspondingly scaled problemsize in 2D. The problem size is changed from 410,881 to13,148,192 unknowns. The scaled speedup data and a linearcurve fitted to it based on Gustafson’s law [13] are depictedin Fig. 8. The fitted value for s is 0.12which is different fromthat given by Amdah’s law and strong scaling. The discrep-ancy in s can be due to the approximations in both Amdah’sand Gustafson’s laws, i.e., the serial part is assumed to re-main constant, and the parallel part is assumed to be scaledup in proportion to the number of threads.

In our strong and weak scaling studies, no schedulingtype is specified for the parallel loops, i.e., OpenMP uses itsdefault scheduling type. In Fig. 9 a comparison betweenstatic and dynamic scheduling with different chunk sizes isprovided. The number of threads is chosen to be 12 and 16,

Figure 7: Strong scaling of the multi-physics NR method interms of speedup with respect to 1 core implementation. Thedashed line is the fitted curve based on Amdah’s law.

Figure 8: Weak scaling of the multi-physics NR method interms of scaled speedup with respect to 1 core implementation.The dashed line is the fitted curve based on Gustafson’s law

Figure 9: Speedup of the parallel NR method in terms ofOpenMP for loop’s scheduling type and chunk size when thenumber of threads = 12 and 16.

when the problem size is the same as that of strong scaling,i.e., 4,173,281 unknowns in 3D.According to Fig. 9, the bestperformance observed is similar to that delivered by the de-fault scheduling in Fig. 7. Also, for reasonably large chunksizes, the difference between static and dynamic schedulingis insignificant, which is due to the fact that the iterations ofthe parallel loops in the code have almost the same compu-tational work.

We compare our OpenMP NR algorithm to a parallelimplementation of the traditional NR method provided bythe optimized library PETSc (Portable, Extensible Toolkitfor Scientific Computation)[4, 5, 6]. PETSc employs theMessage Passing Interface (MPI) standard for communica-

: Preprint submitted to Elsevier Page 7 of 10

Figure 10: Execution times using 16 cores.

tion between parallel tasks. The PETSc implementation istested on a single cluster node with total 40 number of cores,where one MPI task is defined per core. The Jacobian ma-trix is formed at each NR iteration, when the assembly ofthe Jacobian is done in parallel using deal.II’s WorkStreamshared-memory model. For the solving stage, PETSc pro-vides an interface to a variety of iterative and direct solvers,fromwhich theMUltifrontalMassively Parallel sparse directSolver (MUMPS) [2, 1] is selected here. Figure10 showsthe average execution times per time step for our OpenMPNR code with 16 threads, and the PETSc implementationwith 16 threads for the assembly and 16 MPI tasks for thesolution, respectively. Problem sizes change from 500K to33M unknowns in 3D. The Parallel NR demonstrated fasterexecution time while preserving linear scalability with thenumber of unknowns. As the problem size increases, theoverhead due to PETSc’s MPI calls reduces resulting in im-proved efficiency for larger problems.

Finally, the convergence plot of the parallel NR methodis depicted and compared to that of a Gauss-Seidel algorithmwhen applied to the same electrical-thermal problem in Fig.11. As for the NR approach, in the early iterations, changesof the residual norm are almost linear; however, when we getsufficiently close to the solution, the quadratic convergencecan be observed in the last three iterations until the conver-gence criterion, i.e., a residual norm smaller than 10−10, isachieved.

6. ConclusionsThe message passing properties of the Belief Propaga-

tion algorithm on a graphical model are exploited for thefirst time to compute and distribute local sensitivities insteadof forming a large Jacobian matrix in a multi-physics sce-nario. This approach provides a highly parallel NR algo-rithm for solving the coupled electrical-thermal problem inRFAmodeling. This problem has often been over-simplifiedin the literature because of its high computational cost. Asimplified model of the coupled problem that is solved withthe new NR algorithm. In order to verify the correctnessof the algorithm, obtained results are compared with COM-SOL multiphysics simulation, showing good fidelity. Theparallel scalability of the FGaBP method is retained in thenew multi-physics version proposed here. Our results show

Figure 11: The convergence plot of the parallel NR algorithmcompared to that of a Gauss-Seidel algorithm.

speedups of sixteen times with respect to one CPU core.Shared-memory implementation of our algorithms showedfaster run time against highly optimized open-source libraryPETSc. Future work should include comparison against anoptimized multi-threaded solver such as Parallel Sparse Di-rect Solver (PARDISO) [21, 22], distributed-memory imple-mentation based on MPI on cluster architectures, compari-sonwith parallel implementations of inexact Newton’smeth-ods including JFNKmethods, and the application of the pro-posed method to more realistic models of RFA.

7. AcknowledgmentsThis research was enabled in part by support provided

by Calcul Québec (www.calculquebec.ca), Compute Canada(www.computecanada.ca), and also financial support fromNSERC (The Natural Sciences and Engineering ResearchCouncil of Canada).

A. AppendixA local Jacobian matrix is constructed for FNa, based onthe sensitivities calculated during FGaBP:

J(m)a =

[

I2nc×2nc J(m)a,vJ(m)a,T Inc×nc .

]

(25)

The off-diagonal elements of the local Jacobin matrix areupdated during FGaBP iterations; inside iterationm of New-ton’s method and iteration t of FGaBP we have:

J(m,t)a,v (k, l) =)�(m,t)ak,v

)�(m−1)al,T

=)�(m,t)ak,v

)T (m−1)l

)T (m−1)l

)�(m−1)al,T

= 1�(m−1)l,T

)�(m,t)ak,v

)T (m−1)l

,

(26)for 0 ≤ k < nc, and:

J(m,t)a,v (k, l) =)�(m,t)ak,v

)�(m−1)al,T

=)�(m,t)ak,v

)T (m−1)l

)T (m−1)l

)�(m−1)al,T

= 1�(m−1)l,T

)�(m,t)ak,v

)T (m−1)l

,

(27)for nc ≤ k < 2nc. Here the chain rule and also the fact thatbased on (13) we can write )Tl∕)�al,T = 1∕�l,T are used.

: Preprint submitted to Elsevier Page 8 of 10

Now, based on (21) the elements of the sub-matrix J(m,t)a,v areupdated as:

J(m,t)a,v (k, l) =1

�(m−1)l,T

)fk)T (m−1)l

, (28)

for 0 ≤ k < nc, and:

J(m,t)a,v (k, l) =1

�(m−1)l,T

( )gk)T (m−1)l

+∑

l′∈ (a)

)gk)v(m,t∗)

l′

)v(m,t∗)l′

)T (m−1)l

)

,

(29)for nc ≤ k < 2nc, where:)fk

)T (m−1)l

= −1W̃ 2

k,v

)W̃k,v

)T (m−1)l

, (30)

)gk)T (m−1)l

= −1W̃ 2

k,v(K̄v

(m,t∗))T ⋅ C̃v)W̃k,v

)T (m−1)l

+

(K̄v(m,t∗))T

W̃k,v⋅

)C̃v)T (m−1)l

, (31)

and

)gk)v(m,t∗)

l′

=

⎧

⎪

⎨

⎪

⎩

0 k = l′

1W̃k,v(Tl)

)(

K̄v(m,t∗)

)T

)v(m,t∗)l′

⋅ C̃v(Tl) k ≠ l′ .(32)

Finally, the partial derivatives in 30 to 32 are calculated an-alytically based on (6) to (10).

Following a similar approach, the elements of the sub-matrix J(m,t)a,T can be calculated as follows:

J(m,t)a,T (k, l) =)�(m,t)ak,T

)�(m−1)al,v

=)�(m,t)ak,T

)v(m−1)l

)v(m−1)l

)�(m−1)al,v

= 1�(m−1)l,v

)�(m,t)ak,T

)v(m−1)l

= 1�(m−1)l,v

( )ℎk)v(m−1)l

+∑

l′∈ (a)

)ℎk)T (m)

l′

)T (m)l′

)v(m−1)l

)

,

(33)for 0 ≤ l < nc, and:

J(m,t)a,T (k, l) =)�(m,t)ak,T

)�(m−1)al,v

=)�(m,t)ak,T

)v(m−1)l

)v(m−1)l

)�(m−1)al,v

=−�(m−1)l,v

[�(m−1)l,v ]2

)�(m,t)ak,T

)v(m−1)l

=−�(m−1)l,v

[�(m−1)l,v ]2( )ℎk)v(m−1)l

+∑

l′∈ (a)

)ℎk)T (m)

l′

)T (m)l′

)v(m−1)l

)

,

(34)for for nc ≤ l < 2nc. Here, the chain rule and the followingrelation are used:vl =

�l,v�l,v

= 1�l,v

∑

a′∈ (l)

�a′ l,v , (35)

whichmeans )vl∕)�al,v = 1∕�l,v and )vl∕)�al,v = −�l,v∕�2l,v.Similar to 30 to 32, the sensitivities )ℎk∕)Tl′ and )ℎk∕)vlare calculated as follows:

)ℎk)T (m)

l′

=

⎧

⎪

⎨

⎪

⎩

0 k = l′

1W̃k,v

)(

K̄T(m,t∗)

)T

)T (m)l′

⋅ C̃T k ≠ l, (36)

)ℎk)v(m−1)l

=)Bk,T)v(m−1)l

+ 1W̃k,v

)(

K̄T(t∗))T

)v(m−1)l

C̃T , (37)

in which the partial derivatives are calculated analyticallybased on (6) to (10).

References[1] Amestoy, P., Buttari, A., L’Excellent, J.Y., Mary, T., 2019. Perfor-

mance and Scalability of the Block Low-RankMultifrontal Factoriza-tion on Multicore Architectures. ACM Transactions on MathematicalSoftware 45, 2:1–2:26.

[2] Amestoy, P., Duff, I.S., Koster, J., L’Excellent, J.Y., 2001. A fullyasynchronous multifrontal solver using distributed dynamic schedul-ing. SIAM Journal on Matrix Analysis and Applications 23, 15–41.

[3] Audigier, C., Mansi, T., Delingette, H., Rapaka, S., Mihalef, V.,Carnegie, D., Boctor, E., Choti, M., Kamen, A., Ayache, N., et al.,2015. Efficient lattice boltzmann solver for patient-specific radiofre-quency ablation of hepatic tumors. IEEE Transactions on MedicalImaging 34, 1576–1589.

[4] Balay, S., Abhyankar, S., Adams, M.F., Brown, J., Brune, P., Buschel-man, K., Dalcin, L., Dener, A., Eijkhout, V., Gropp, W.D., Karpeyev,D., Kaushik, D., Knepley, M.G., May, D.A., McInnes, L.C., Mills,R.T., Munson, T., Rupp, K., Sanan, P., Smith, B.F., Zampini, S.,Zhang, H., Zhang, H., 2020. PETSc Users Manual. Technical Re-port ANL-95/11 - Revision 3.13. Argonne National Laboratory. URL:https://www.mcs.anl.gov/petsc.

[5] Balay, S., Abhyankar, S., Adams, M.F., Brown, J., Brune, P., Buschel-man, K., Dalcin, L., Dener, A., Eijkhout, V., Gropp, W.D., Karpeyev,D., Kaushik, D., Knepley, M.G., May, D.A., McInnes, L.C., Mills,R.T., Munson, T., Rupp, K., Sanan, P., Smith, B.F., Zampini, S.,Zhang, H., Zhang, H., 2019. PETSc Web page. https://www.mcs.

anl.gov/petsc. URL: https://www.mcs.anl.gov/petsc.[6] Balay, S., Gropp, W.D., McInnes, L.C., Smith, B.F., 1997. Efficient

management of parallelism in object oriented numerical software li-braries, in: Arge, E., Bruaset, A.M., Langtangen, H.P. (Eds.), ModernSoftware Tools in Scientific Computing, Birkhäuser Press. pp. 163–202.

[7] Bangerth, W., Hartmann, R., Kanschat, G., 2007. deal. ii—a general-purpose object-oriented finite element library. ACM Transactions onMathematical Software (TOMS) 33, 24–es.

[8] Berjano, E.J., 2006. Theoretical modeling for radiofrequency abla-tion: state-of-the-art and challenges for the future. BioMed. Eng. On-line 5.

[9] Cheong, J.K., Yap, S., Ooi, E.T., Ooi, E.H., 2019. A computationalmodel to investigate the influence of electrode lengths on the singleprobe bipolar radiofrequency ablation of the liver. Computer methodsand programs in biomedicine 176, 17–32.

[10] El-Kurdi, Y., Dehnavi, M.M., Gross, W.J., Giannacopoulos, D., 2015.Parallel finite element technique using gaussian belief propagation.Comput. Phys. Commun. 193, 38–48.

[11] El-Kurdi, Y., Gross, W.J., Giannacopoulos, D., 2012. Efficient imple-mentation of gaussian belief propagation solver for large sparse diag-onally dominant linear systems. IEEE Trans. Magn. 48, 471–474.

[12] El-Kurdi, Y., Gross, W.J., Giannacopoulos, D., 2014. Parallel multi-grid acceleration for the finite-element gaussian belief propagation al-gorithm. IEEE transactions on magnetics 50, 581–584.

: Preprint submitted to Elsevier Page 9 of 10

[13] Gustafson, J.L., 1988. Reevaluating amdahl’s law. Communicationsof the ACM 31, 532–533.

[14] Hall, S.K., Ooi, E.H., Payne, S.J., 2015. Cell death, perfusion andelectrical parameters are critical in models of hepatic radiofrequencyablation. International Journal of Hyperthermia 31, 538–550.

[15] Ho, S., Niu, S., Fu, W., 2012. Design and analysis of novel focusedhyperthermia devices. IEEE Trans. Magn. 48, 3254–3257.

[16] Keyes, D.E., McInnes, L.C., Woodward, C., Gropp, W., Myra, E.,Pernice, M., et al., 2013. Multiphysics simulations: Challenges andopportunities. The International Journal of High Performance Com-puting Applications 27, 4–83.

[17] Mariappan, P., Weir, P., Flanagan, R., Voglreiter, P., Alhonnoro, T.,Pollari, M., Moche, M., Busse, H., Futterer, J., Portugaller, H.R.,et al., 2017. Gpu-based rfa simulation for minimally invasive cancertreatment of liver tumours. International journal of computer assistedradiology and surgery 12, 59–68.

[18] Payne, S., Flanagan, R., Pollari, M., Alhonnoro, T., Bost, C., O’Neill,D., Peng, T., Stiegler, P., 2011. Image-based multi-scale modellingand validation of radio-frequency ablation in liver tumours. Phil.Trans. R. Soc. A 369, 4233–4254.

[19] Pennes, H.H., 1998. Analysis of tissue and arterial blood temperaturesin the resting human forearm. Journal of applied physiology 85, 5–34.

[20] Pérez, J.J., Pérez-Cajaraville, J.J., Muñoz, V., Berjano, E., 2014.Computer modeling of electrical and thermal performance duringbipolar pulsed radiofrequency for pain relief. Medical physics 41,071708.

[21] Petra, C.G., Schenk, O., Anitescu, M., 2014a. Real-time stochasticoptimization of complex energy systems on high-performance com-puters. Computing in Science & Engineering 16, 32–42.

[22] Petra, C.G., Schenk, O., Lubin, M., Gärtner, K., 2014b. An aug-mented incomplete factorization approach for computing the schurcomplement in stochastic optimization. SIAM Journal on ScientificComputing 36, C139–C162.

[23] Reinhardt, M., Brandmaier, P., Seider, D., Kolesnik, M., Jenniskens,S., Sequeiros, R.B., Eibisberger, M., Voglreiter, P., Flanagan, R.,Mariappan, P., et al., 2017. A prospective development study ofsoftware-guided radio-frequency ablation of primary and secondaryliver tumors: Clinical intervention modelling, planning and proof forablation cancer treatment (clinicimppact). Contemporary clinical tri-als communications 8, 25–32.

[24] Rodgers, D.P., 1985. Improvements in multiprocessor system design.ACM SIGARCH Computer Architecture News 13, 225–231.

[25] Solazzo, S.A., Liu, Z., Lobo, S.M., Ahmed, M., Hines-Peralta, A.U.,Lenkinski, R.E., Goldberg, S.N., 2005. Radiofrequency ablation: im-portance of background tissue electrical conductivity—an agar phan-tom and computer modeling study. Radiology 236, 495–502.

[26] Szabó, T., 2008. On the discretization time-step in the finite elementtheta-method of the discrete heat equation, in: International Confer-ence on Numerical Analysis and Its Applications, Springer. pp. 564–571.

[27] Tang, Y., Jin, T., Flesch, R.C., 2017. Numerical temperature analy-sis of magnetic hyperthermia considering nanoparticle clustering andblood vessels. IEEE Trans. Magn. 53, 1–6.

[28] Trujillo, M., Berjano, E., 2013. Review of the mathematical functionsused to model the temperature dependence of electrical and thermalconductivities of biological tissue in radiofrequency ablation. Inter-national Journal of Hyperthermia 29, 590–597.

[29] Voglreiter, P., Mariappan, P., Pollari, M., Flanagan, R., Sequeiros,R.B., Portugaller, R.H., Fütterer, J., Schmalstieg, D., Kolesnik, M.,Moche, M., 2018. Rfa guardian: Comprehensive simulation of ra-diofrequency ablation treatment of liver tumors. Scientific reports 8,787.

: Preprint submitted to Elsevier Page 10 of 10