Simulation Modelling Practice and Theory 17 (2009) 1603–1617

Contents lists available at ScienceDirect

Simulation Modelling Practice and Theory

journal homepage: www.elsevier .com/ locate/s impat

Simulation and verification of Zhang neural network for onlinetime-varying matrix inversion

Yunong Zhang *, Chenfu Yi, Weimu MaSchool of Information Science and Technology, Sun Yat-Sen University, Guangzhou 510275, China

a r t i c l e i n f o

Article history:Received 18 January 2009Received in revised form 3 June 2009Accepted 3 July 2009Available online 9 July 2009

Keywords:Simulation and verificationRecurrent neural networkTime-varying matrix inversionKronecker productSymbolic math

1569-190X/$ - see front matter � 2009 Elsevier B.Vdoi:10.1016/j.simpat.2009.07.001

* Corresponding author. Tel./fax: +86 20 8411359E-mail address: ynzhang@ieee.org (Y. Zhang).

a b s t r a c t

Differing from gradient-based neural networks (GNN), a special kind of recurrent neuralnetwork has recently been proposed by Zhang et al. for real-time inversion of time-varyingmatrices. The design of such a recurrent neural network is based on a matrix-valued errorfunction instead of a scalar-valued norm-based energy-function. In addition, it is depictedin an implicit dynamics instead of an explicit dynamics. This paper investigates the simu-lation and verification of such a Zhang neural network (ZNN). Four important simulationtechniques are employed to simulate this system: (1) Kronecker product of matrices isintroduced to transform a matrix-differential-equation (MDE) to a vector differential equa-tion (VDE) [i.e., finally, there is a standard ordinary-differential-equation (ODE) formula-tion]. (2) MATLAB routine ‘‘ode45” with a mass-matrix property is introduced tosimulate the transformed initial-value implicit ODE system. (3) Matrix derivatives areobtained using the routine ‘‘diff” and symbolic math toolbox. (4) Various implementationerrors and different types of activation functions are investigated, further demonstratingthe advantages of the ZNN model. Three illustrative computer-simulation examples sub-stantiate the theoretical results and efficacy of the ZNN model for online time-varyingmatrix inversion.

� 2009 Elsevier B.V. All rights reserved.

1. Introduction

Online matrix-inversion problems are widely encountered in science and engineering fields, e.g., in optimization [1,2],signal-processing [3,4], electromagnetic systems [5], robot control [6,7], statistics [4,8], and physics [9].

There are two general types of solutions to the matrix-inversion problem. One type of solution is the numerical algo-rithms performed on digital computers (i.e., on today’s computers). Usually, the minimal arithmetic operations for a ma-trix-inversion algorithm are proportional to the cube of the matrix dimension n, i.e., Oðn3Þ operations. Consequently, suchserial-processing numerical algorithms may not be efficient enough for large-scale online or real-time applications. To rem-edy this computational problem, some Oðn2Þ-operation algorithms were proposed (e.g., in [4,8]); however, they may still notbe fast enough [4]. Many parallel processing methods, the second type of matrix inversion, have been proposed, analyzed,and implemented on specific architectures [3,6,7,10–20].

The dynamic system approach has been an important parallel processing method for solving matrix-inversion problems[3,6,10–15,21]. Recently, due to in-depth research in neural networks, numerous dynamic and analog solvers based on recur-rent neural networks (RNN) have been developed and investigated [3,6,10–15,21–28]. Thus, the neural-dynamic approach isnow regarded as a powerful alternative for online computation in view of its parallel distributed nature and the convenienceof circuits implementation [6,7,16,20,29].

. All rights reserved.

7.

1604 Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617

In March 2001 [10–13], Zhang et al. proposed a special kind of RNN for the real-time inversion of time-varying matrices,differing from gradient-based neural networks (GNN) for constant matrix inversion [6,14–16]. To solve for a time-varyingmatrix inverse A�1ðtÞ 2 Rn�n, the design of the neural network is based on the defining equation, AðtÞXðtÞ � I ¼ 0, over timet 2 ½0;þ1Þ, where AðtÞ 2 Rn�n is a smoothly time-varying nonsingular matrix, and I denotes an appropriately-dimensionedidentity matrix. As proposed and investigated in [10–13,21–25], the following recurrent neural network [simply termed,Zhang neural network (ZNN), for ease of presentation] can be used to solve for the time-varying matrix inverse A�1ðtÞ in realtime t:

AðtÞ _XðtÞ ¼ � _AðtÞXðtÞ � CFðAðtÞXðtÞ � IÞ ð1Þ

where

� XðtÞ 2 Rn�n, starting from initial condition Xð0Þ 2 Rn�n, is the activation state matrix corresponding to the theoreticalinverse A�1ðtÞ;

� the matrix-valued parameter C 2 Rn�n could simply be cI with c > 0 2 R;� _AðtÞ 2 Rn�n denotes the derivative of matrix AðtÞ with respect to time t [or simply termed, the time derivative of matrix

AðtÞ]; and,� Fð�Þ : Rn�n ! Rn�n denotes an activation-function array of the neural network, of which a simple example is the linear one,

i.e., FðAðtÞXðtÞ � IÞ ¼ AðtÞXðtÞ � I.

As compared to gradient-based recurrent neural networks, the difference and novelty of the ZNN model lie in the follow-ing facts.

(1) The design of ZNN model (1) is based on the elimination of every entry eijðtÞ of the matrix-valued error functionEðtÞ ¼ AðtÞXðtÞ � I, where i; j 2 f1;2; . . . ;ng. In contrast, the design of the GNN model

_XðtÞ ¼ �cATFðAXðtÞ � IÞ ð2Þ

is based on the elimination of the scalar-valued norm-based energy function EðtÞ ¼ kAXðtÞ � Ik2 (note that, in such aGNN design, matrix A could only be constant) [6,14–16].

(2) ZNN model (1) is depicted in an implicit dynamics, i.e., AðtÞ _XðtÞ ¼ � � �, which coincides well with systems in natureand in practice (e.g., with analogue electronic circuits and mechanical systems, owing to Kirchhoff’s and Newton’slaws, respectively [10–13,21–25]). Evidently, with simple operations, the implicit systems can be transformed intoexplicit systems [13,30], if necessary. In contrast, GNN model (2) is depicted in an explicit dynamics, i.e., _XðtÞ ¼ � � �,which is usually associated with conventional Hopfield-type and/or gradient-based neural networks [6,14–16].Comparing the implicit and explicit dynamic systems, it can be seen that the former could have a greater abilityto preserve more physical parameters, e.g., even in the coefficient matrix on the left hand side of the system [i.e.,AðtÞ in (1)].

(3) ZNN model (1) could systematically and methodologically exploit the time-derivative information (or say, trend) ofcoefficient matrix AðtÞ during its real-time inverting process. This appears to be an important reason why the neuralstate XðtÞ of ZNN model (1) could globally converge to the exact inverse of a time-varying matrix [12]. On the otherhand, GNN model (2) has not exploited this important information of _AðtÞ, and thus may not be effective enough onthe time-varying matrix inversion.

(4) By making good use of the time-derivative information of matrix AðtÞ, ZNN model (1) belongs to a predictive approach,which is more effective on the system convergence to a time-varying theoretical inverse [10–13]. In contrast, GNNmodel (2) belongs to a tracking approach, which adapts to the change of matrix AðtÞ in a posterior passive manner.

(5) As shown in the following sections, the neural state XðtÞ computed by ZNN model (1) can globally exponentially con-verge to the theoretical time-varying inverse A�1ðtÞ. In contrast, GNN model (2) can only generate the approximationresults of such a theoretical inverse A�1ðtÞ with much larger steady-state errors, thus is less effective on time-varyingmatrix inversion.

To the best of the authors’ knowledge, to date, few studies have been published dealing with the simulation of recurrentneural networks. In this paper, the simulation and verification of ZNN model (1) using MATLAB [31] is investigated. To sim-ulate such a special implicit dynamic system, four important simulation techniques (as briefed below) are employed.

� Kronecker product of matrices is introduced to transform the matrix differential Eq. (1) to a vector differential equation(VDE). That is, finally, there is a standard ordinary differential equation (ODE) formulation.

� MATLAB routine ‘‘ode45” with a mass-matrix property is introduced to simulate the transformed initial-value ODE sys-tem, where matrix AðtÞ on the left hand side of ZNN model (1) is termed a mass matrix.

� Matrix derivatives [e.g., _AðtÞ] are obtained using MATLAB routine ‘‘diff” and symbolic math toolbox [31].� In addition to various implementation errors, different types of activation-function arrays Fð�Þ are coded and simulated in

order to illustrate the characteristics of ZNN model (1) working in different situations.

Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617 1605

Three illustrative computer-simulation examples further substantiate the theoretical results and efficacy of ZNN model(1) for online time-varying matrix inversion. The rest of this paper is organized as follows. In Section 2, the theoretical resultsfrom ZNN model (1) are presented, especially focusing on the network convergence and robustness. In Section 3, four MAT-LAB coding and simulation techniques are provided for ZNN model (1) which solves for A�1ðtÞ in general situations. In Section4, three illustrative examples (i.e., a sinusoidal-matrix inversion, a Toeplitz matrix inversion, and a constant matrix inver-sion) are presented to show the network convergence and robustness. Concluding remarks are given in Section 5.

2. Theoretical results

Before presenting the theoretical results, a description of the design parameters appearing in ZNN model (1) is presented.Similar to the conventional neural-network approaches, the matrix parameter C ¼ cI in Eq. (1), like a set of inductanceparameters or reciprocals of capacitance parameters in electronic circuits, is set as large as the hardware permits [29] or se-lected appropriately for experimental and/or simulative purposes. Moreover, it follows from ZNN model (1) that differentdesign parameter values and activation-function arrays Fð�Þ lead to different ZNN performance. In general, any monotoni-cally-increasing odd function can be chosen as the activation function of the neural network. Since March 2001 [21], fourtypes of activation functions (i.e., linear activation function, power activation function, sigmoid activation function andpower-sigmoid function) have been introduced and used for the proposed ZNN models (for more details, see [10–13,21–25]).

From the analysis results of [10–13,21–25], the following general observations on global convergence and exponentialconvergence are summarized and presented for ZNN model (1).

Lemma 1. Given a smoothly time-varying nonsingular matrix AðtÞ 2 Rn�n, if a monotonically-increasing odd activation-functionarray Fð�Þ is used, then the state matrix XðtÞ of ZNN model (1), starting from any initial state Xð0Þ, will always converge to thetime-varying theoretical inverse A�1ðtÞ. In addition, more detailed properties of ZNN model (1) can be referred to in [10–13,21–25].

The analog implementation or simulation of recurrent neural networks is usually assumed to be under ideal conditions.However, there are always some realization errors involved. For example, the matrix-implementation error, differentiationerror, and the model-implementation error appear frequently in hardware simulation and realization. For the appearance ofthese errors in ZNN model (1), the following lemma is used:

Lemma 2 ([10–12,21,23]). Consider the perturbed ZNN model A _X ¼ � _AX � cFðAX � IÞ with imprecise matrix-implementationA ¼ Aþ DA. If kDAðtÞk 6 e1 for any time instant t 2 ½0;1Þ, 90 6 e1 < þ1, then the steady-state computational errorlimt!1kXðtÞ � A�1ðtÞk is uniformly upper bounded by a positive scalar, provided that the invertibility of matrix A still holdstrue.

For the differentiation error and the model-implementation error, the following dynamics is considered, as compared tothe original dynamic Eq. (1).

AðtÞ _XðtÞ ¼ �ð _AðtÞ þ DBðtÞÞXðtÞ � cFðAðtÞXðtÞ � IÞ þ DCðtÞ ð3Þ

where DBðtÞ 2 Rn�n and DCðtÞ 2 Rn�n, respectively denote the differentiation error and the model-implementation error,which result from truncating/roundoff errors in digital realization and/or high-order residual errors of circuit componentsin analog realization. Generally speaking, if kDBðtÞk 6 e2 and kDCðtÞk 6 e3 for any t 2 ½0;1Þ, then the computational errorkXðtÞ � A�1ðtÞk is uniformly upper bounded by some scalar, and its steady-state error could be expressed as in Theorem 2of [11]. More detailed robustness properties of ZNN model (3) can be seen in [10–12,21,23].

The robustness study could give us more insights on how to handle the problem of imperfect implementation of the pre-sented ZNN model (1) and/or other related issues on neural network realization. In summary, we need to:

� implement matrix A as accurately as possible,� use a large value of design parameter c if the hardware permits, and� use the activation-function nonlinearity (e.g., using power-sigmoid activation functions) to remedy hardware-implemen-

tation errors.

3. Simulation techniques

While Section 2 presents the main theoretical results of the presented ZNN model (1), the following MATLAB simulationtechniques [31] are investigated in this section to show the characteristics of such a ZNN model.

3.1. Coding of activation function

To simulate the general matrix-inversion ZNN model (1), the activation functions are to be designed and coded firstly inMATLAB. Inside the body of a user-defined function, the routine ‘‘nargin” returns the number of input arguments, which are

1606 Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617

used to call the function. Using ‘‘nargin”, different types of activation functions can be generated with their default inputargument(s). For example, the power-sigmoid activation-function array as defined in [10–13,21–25] with n ¼ 4 and p ¼ 3being their default values can be coded and generated as in Appendix A (I).

3.2. Kronecker product and vectorization

Review ZNN model (1) and GNN model (2). Their dynamic equations are all described in matrix-form, which cannot bedirectly simulated. Thus, the Kronecker product and vectorization techniques are needed to transform such matrix-form dif-ferential equations to vector-form differential equations.

� In a general situation, for the given matrices A ¼ ½aij� 2 Rm�n and B ¼ ½bij� 2 Rp�q, the Kronecker product of A and B isdenoted by A� B and defined as the following block matrix [32]:

A� B :¼a11B � � � a1nB

..

. ... ..

.

am1B � � � amnB

0BB@

1CCA 2 Rmp�nq;

which is also known as the direct product or tensor product. Note that, in general, A� B–B� A; specifically, for our case,I � A ¼ diagðA; . . . ;AÞ.

� In a general situation, given X ¼ ½xij� 2 Rn�s, it is possible to vectorize X as a column vector, vecðXÞ 2 Rns�1, which is definedas

vecðXÞ :¼ ½x11; . . . ; xn1; x12; . . . ; xn2; . . . ; x1s; . . . ; xns�T :

As stated in [32], in a general situation, given A 2 Rm�n and B 2 Rm�s, with X unknown, the matrix equation AX = B could bevectorized as the matrix-vector equation ðI � AÞvecðXÞ ¼ vecðBÞ.

Now, based on the above Kronecker product and vectorization techniques, for simulation proposes, let us transform thematrix differential Eq. (1) to a vector differential equation. The following theorem is thus obtained:

Theorem. Matrix differential Eq. (1) can be reformulated as the following vector differential equation:

ðI � AÞvecð _XÞ ¼ �ðI � _AÞvecðXÞ � ðI � CÞF ðI � AÞvecðXÞ � vecðIÞð Þ; ð4Þ

or, simply put,

M _x ¼ � _Mx� cF Mx� vecðIÞð Þ;

where the activation-function array Fð�Þ in (4) is defined the same as in (1), except that its dimensions are changed hereafter asFð�Þ : Rn2�1 ! Rn2�1, the so-called mass matrix MðtÞ :¼ I � AðtÞ, and the integration vector x :¼ vecðXÞ.

Proof. For the reader’s convenience, the matrix-form differential Eq. (1) is repeated here: AðtÞ _XðtÞ ¼� _AðtÞXðtÞ � CFðAðtÞXðtÞ � IÞ. By vectorizing Eq. (1) based on the Kronecker product and the above vecð�) operator, the lefthand side of ZNN model (1) becomes

vec AðtÞ _XðtÞ� �

¼ ðI � AðtÞÞvecð _XðtÞÞ: ð5Þ

The right hand side of ZNN model (1) correspondingly becomes

vecð� _AðtÞXðtÞ � CFðAðtÞXðtÞ � IÞÞ ¼ vec � _AðtÞXðtÞ� �

þ vec �CFðAðtÞXðtÞ � IÞð Þ

¼ �vec _AðtÞXðtÞ� �

� vec CFðAðtÞXðtÞ � IÞð Þ

¼ � I � _AðtÞ� �

vec XðtÞð Þ � ðI � CÞvecðFðAðtÞXðtÞ � IÞÞ: ð6Þ

Note that, as shown in Sub Section 3.1, the definition and coding of the activation-function array Fð�Þ are very flexible andcould also be those of a vectorized array from Rn2�1 to Rn2�1. We thus have

ðI � CÞvecðFðAðtÞXðtÞ � IÞÞ ¼ ðI � CÞFðvecðAðtÞXðtÞ � IÞÞ ¼ ðI � CÞFðvecðAXÞ þ vecð�IÞÞ¼ ðI � CÞFððI � AÞvecðXÞ � vecðIÞÞ: ð7Þ

Combining Eqs. (6) and (7) yields the vectorization result of the right hand side of matrix differential Eq. (1):

vecð� _AðtÞXðtÞ � CFðAðtÞXðtÞ � IÞÞ ¼ �ðI � _AÞvecðXÞ � ðI � CÞFððI � AÞvecðXÞ � vecðIÞÞ: ð8Þ

Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617 1607

Evidently, the vectorization of both sides of matrix differential Eq. (1) should be equal; i.e., Eq. (5) should be equal to Eq. (8).Hence, the presented ZNN model (1) can be transformed to the vector-form (4). In addition, if we define matrixMðtÞ :¼ I � AðtÞ and consider C ¼ cI for the present research, then vector-form (4) could be further simplified as

M _x ¼ � _Mx� cFðMx� vecðIÞÞ:

The proof is thus complete. h

Remarks. The Kronecker product can be generated readily by using routine ‘‘kron”; in other words, A� B can be generatedby code kron(A,B). To generate vec(X), the routine ‘‘reshape” can be used. Specifically, if the matrix X has m rows and ncolumns, then the code of vectorizing X is reshape(X,m*n,1) which generates a column vector,vecðXÞ ¼ ½x11; . . . ; xm1; x12; . . . ; xm2; . . . ; x1n; . . . ; xmn�T .

Based on routines ‘‘kron” and ‘‘vec”, the code in Appendix A (II) could be used to define a function which evaluates theright hand side of vector-form ZNN model (4). In other words, it returns the evaluation of gðt; xÞ in equation M _x ¼ gðt; xÞ,where, in our case, gðt; xÞ :¼ � _Mx� cFðMx� vecðIÞÞ.

3.3. ODE with mass matrix

In the simulation of ZNN model (1), the routine ‘‘ode45” is preferred because ‘‘ode45” can solve the initial-value ODEproblem with a mass matrix, e.g., Mðt; xÞ _x ¼ gðt; xÞ, where the nonsingular matrix Mðt; xÞ on the left hand side of such anequation is termed the mass matrix, and gðt; xÞ :¼ � _Mx� cFðMx� vecðIÞÞ for our case (see the above-presented theorem).

The code in Appendix A (III) could be used to generate a mass matrix Mðt; xÞ :¼ I � AðtÞ in Eq. (4), where a two-dimen-sional time-varying matrix AðtÞ is defined to be ½sin t; cos t;� cos t; sin t�, as an example.

To solve the ODE with a mass matrix, the routine ‘‘odeset” should also be used. Its ‘‘Mass” property should be assigned tobe the function handle @MatrixM, which returns the value of the mass matrix Mðt; xÞ. Note that, if Mðt; xÞ does not depend onthe state variable x and if the function ‘‘MatrixM” is to be invoked with only one input argument t, then the value of the‘‘MStateDep” property of routine ‘‘odeset” could be set as ‘‘none”. Thus, for example, the code in Appendix A (IV) could beused to solve an initial-value ODE problem with state-independent mass matrix MðtÞ and random initial state xð0Þ, where,as shown in Appendix A (II), the function ‘‘ZnnRightHandSide” returns the value of gðt; xÞ of the differential equationMðt; xÞ _x ¼ gðt; xÞ. Specifically, in our case, it returns the evaluation of the right hand side of the vector-form ZNN model (4).

3.4. Obtaining matrix derivative

In the time-varying matrix inversion using ZNN model (1), the time derivative _AðtÞ is assumed to be known or measur-able. This implies that _AðtÞ can be given directly in an analytical form or can be estimated via finite difference. Similarly, inthe vector-form ZNN model (4), the time derivative _MðtÞ ¼ I � _AðtÞ is also needed (i.e., known or measurable). Thus for thesimulation, the routine ‘‘diff” is introduced, as follows, in order to automatically generate the time derivative _MðtÞ and/or_AðtÞ.

� For a vector or matrix X, diff(X) returns the row difference of X. For example, for a matrix X ¼ ½3;7;5; 0;9;2�, diff(X) is[�3,2,�3].

� For a symbolic expression X, diff(X) differentiates X with respect to its free variable as determined by routine ‘‘findsym”.For example, the derivative of sin x2 with respect to variable x can be obtained by using the following commands:x=sym(‘x’); diff(sin(x

2)). The result of such commands is thus 2*cos(x

2)*x.

As can be seen in the code of the function ‘‘ZnnRightHandSide” presented in Appendix A (II), in order to get the time deriv-ative _MðtÞ of MðtÞ, a symbolic argument ‘‘u” can be constructed and then the command D=diff(MatrixM(u)) can be usedto generate _MðtÞ. Note that, without using the above symbolic ‘‘u”, row differences of MatrixM(t) are generated by callingthe command diff(MatrixM(t)), which is not the desired time derivative of MðtÞ.

4. Illustrative examples

In the previous sections, the neural-dynamic models [i.e., ZNN model (1) and GNN model (2)], their theoretical results,and simulation techniques [31] have been presented. For illustration and simulation purposes, in the first example, bothZNN model (1) and GNN model (2) are exploited to solve online for a time-varying sinusoidal-matrix inverse. Two otherillustrative examples are now presented (i.e., a time-varying Toeplitz matrix inversion and a constant matrix inversion) tofurther substantiate the efficacy of the ZNN model on time-varying and constant matrices inversion.

Example 1. Let us consider the following time-varying matrix AðtÞ and its time derivative _AðtÞ:

AðtÞ ¼sin t cos t

� cos t sin t

� �; _AðtÞ ¼

cos t � sin t

sin t cos t

� �:

1608 Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617

Simple manipulations can verify that

A�1ðtÞ ¼ ATðtÞ ¼sin t � cos t

cos t sin t

� �;

which is used to compare the correctness of the neural-network solutions. When used to solve for the above A�1ðtÞ in realtime, ZNN model (1) can take the following specific and simplified form:

sin t cos t

� cos t sin t

� �_x11 _x12

_x21 _x22

� �¼ �

cos t � sin t

sin t cos t

� �x11 x12

x21 x22

� �� cF

sin t cos t

� cos t sin t

� �x11 x12

x21 x22

� ��

1 00 1

� �� �

where array F is constructed by one of the presented four types of activation functions [10–13,21–25], such as the power-sigmoid functions with n ¼ 4 and p ¼ 3. By using Kronecker product and vectorization techniques, we have

MðtÞ ¼ I � AðtÞ ¼

sin t cos t 0 0� cos t sin t 0 0

0 0 sin t cos t

0 0 � cos t sin t

26664

37775; _MðtÞ ¼ I � _AðtÞ;

x ¼ vecðXÞ ¼ ½x11; x21; x12; x22�T , and vecðIÞ ¼ ½1;0;0;1�T . The above matrix-form differential equation can thus be rewritten asthe following vector-form differential equation, which can be solved directly with ‘‘ode45”:

sin t cos t 0 0� cos t sin t 0 0

0 0 sin t cos t

0 0 � cos t sin t

26664

37775

_x11

_x21

_x12

_x22

26664

37775 ¼ �

cos t � sin t 0 0sin t cos t 0 0

0 0 cos t � sin t

0 0 sin t cos t

26664

37775

x11

x21

x12

x22

26664

37775

� cF

sin t cos t 0 0� cos t sin t 0 0

0 0 sin t cos t

0 0 � cos t sin t

26664

37775

x11

x21

x12

x22

26664

37775�

1001

26664

37775

0BBB@

1CCCA:

4.1. Simulation of convergence

The code presented in Appendix B (I) could be used to simulate ZNN model (1), starting from eight random initial states.Fig. 1, which shows the global exponential convergence of such a neural network, is then generated. In the figure, the the-oretical inverse A�1ðtÞ is denoted by dotted red lines, and the neural-network solutions XðtÞ are denoted by solid blue lines.

The norm of the solution error, kXðtÞ � A�1ðtÞk, can also be used and shown to monitor the network convergence. The re-lated codes are shown in Appendix B (II), i.e., the user-defined functions ‘‘NormError” and ‘‘ZnnNormError”. By calling ‘‘Znn-NormError” twice with different c values, Fig. 2 can be generated. It shows that, starting from eight initial states randomlyselected in [�2,2], the state matrices of the presented neural network (1) all converge to the theoretical inverse A�1ðtÞ, where

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

x11 x12

x21 x22

time t (s)

time t (s)

time t (s)

time t (s)

Fig. 1. Online inversion of time-varying sinusoidal matrix AðtÞ by ZNN model (1).

0 1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

2.5

3

3.5

4

γ = 1

time t (s)

0 1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

2.5

3

3.5

4

γ = 10

time t (s)

Fig. 2. Convergence of computational error kXðtÞ � A�1ðtÞk by ZNN model (1).

Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617 1609

the computational errors kXðtÞ � A�1ðtÞk all converge correspondingly to zero. Such a convergence can be expedited byincreasing c. For example, if c is increased to 103, the convergence time is within 4 milliseconds; and, if c is increased to106, the convergence time is within 4 microseconds.

Note that, for comparison, by applying the above-presented simulation techniques, GNN model (2) can be used to invertthe same time-varying matrix AðtÞ as ZNN model (1) does. Figs. 3 and 4 illustrate the performance of GNN model (2) underthe same simulating conditions. It follows from the GNN-related figures that the GNN state XðtÞ (denoted by solid bluecurves) does not fit well with the theoretical inverse A�1ðtÞ (denoted by dotted red curves). In other words, appreciable com-putational errors always exist between the GNN solution and the theoretical inverse A�1ðtÞ. In addition, it can be seen fromFig. 4 that, by using GNN model (2) for the online time-varying matrix inversion, its computational error kXðtÞ � A�1ðtÞk isquite large. This possibly occurs because GNN model (2) does not make full use of the time-derivative information of matrixAðtÞ.

4.2. Robustness simulation

Similar to the transformation of matrix-form differential Eq. (1) to vector-form differential Eq. (4), the perturbed ZNNmodel (3) can be rewritten as the following vector-form differential equation:

ðI � AÞvecð _XÞ ¼ �ðI � _Aþ I � DBÞvecðXÞ � cF ðI � AÞvecðXÞ � vecðIÞð Þ þ vecðDCÞ ð9Þ

or, simply put,

M _x ¼ �ð _M þ I � DBÞx� cFðMx� vecðIÞÞ þ vecðDCÞ;

where the activation-function array Fð�Þ : Rn2�1 ! Rn2�1, mass matrix MðtÞ :¼ I � AðtÞ, and integration vector x :¼ vecðXÞ.

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

x11 x12

x21 x22

time t (s) time t (s)

time t (s) time t (s)

Fig. 3. Online inversion of time-varying sinusoidal matrix AðtÞ by GNN (2).

0 1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

2.5

3

3.5

4

γ = 1

time t (s)

Fig. 4. Convergence of solution error kXðtÞ � A�1ðtÞk as of GNN model (2).

1610 Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617

To show the robustness characteristics of ZNN model (1), the following differentiation error DBðtÞ and model-implemen-tation error DCðtÞ are added in a higher-frequency sinusoidal form (with e2 ¼ e3 ¼ 0:5):

−

−

−

−

DBðtÞ ¼ e2cos 3t � sin 3t

sin 3t cos 3t

� �; DCðtÞ ¼ e3

sin 3t 00 cos 3t

� �:

The code presented in Appendix B (III) could be used to define the function ‘‘ZnnRightHandSideImprecise” for ODE solvers,which returns the value of the right hand side of the perturbed vector-form differential Eq. (9) [or equivalently, the perturbedmatrix-form differential Eq. (3)]. Based on the function ‘‘ZnnRightHandSideImprecise” and the function defined in AppendixB (IV) (i.e., ‘‘ZnnRobust”), Fig. 5 can be generated using c ¼ 1 and c ¼ 10. As shown in Fig. 5a, with a large differentiation errorDBðtÞ and model-implementation error DCðtÞ, when c ¼ 1, the neural network solution is not very close to the theoretical in-verse A�1ðtÞ. But with a larger value of design parameter c (e.g., 10), as shown in Fig. 5b, the neural-network solution againbecomes very close to the theoretical inverse A�1ðtÞ.

Similarly, the computational error kXðtÞ � A�1ðtÞk of the perturbed ZNN model (3) can be shown in the presence of largedifferentiation and model-implement errors. To do so, in the previously presented function ‘‘NormError”, ‘‘ZnnRightHand-Side” only needs to be changed to ‘‘ZnnRightHandSideImprecise”. See Fig. 6. Even with large realization errors, the compu-tational error kXðtÞ � A�1ðtÞk synthesized by the perturbed neural network (3) is still bounded and very small. Moreover, asthe design parameter c increases from 1 to 100, the convergence is expedited and the steady-state computational error isdecreased. It is observed from other simulation data that when using power-sigmoid activation functions, the maximumsteady-state computational error is 6� 10�3 and 6� 10�4, respectively, for c ¼ 100 and c ¼ 1000. These simulation resultshave substantiated the theoretical results presented in previous sections and in [11].

0 2 4 6 8 102

1

0

1

2

0 2 4 6 8 102

1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

x11 x12

x21 x22

time t (s)

time t (s)

time t (s)

time t (s)

(a) γ = 1

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

0 2 4 6 8 10−2

−1

0

1

2

x11 x12

x21 x22

time t (s)

time t (s)

time t (s)

time t (s)

(b) γ = 10

Fig. 5. Online inversion of sinusoidal matrix AðtÞ by perturbed ZNN model (3).

Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617 1611

Example 2. Let us consider the online inversion of the following time-varying Toeplitz matrix [4]:

0.

1.

2.

−

−

AðtÞ ¼

a1ðtÞ a2ðtÞ a3ðtÞ � � � anðtÞa2ðtÞ a1ðtÞ a2ðtÞ � � � an�1ðtÞ

a3ðtÞ a2ðtÞ a1ðtÞ � � � ...

..

. ... . .

. . ..

a2ðtÞanðtÞ an�1ðtÞ � � � a2ðtÞ a1ðtÞ

2666666664

37777777752 Rn�n;

where aðtÞ :¼ ½a1ðtÞ; a2ðtÞ; a3ðtÞ; . . . ; anðtÞ�T denotes the first column vector of matrix AðtÞ ¼ ½aijðtÞ� 2 Rn�nði; j ¼ 1;2; . . . ; nÞ. Leta1ðtÞ ¼ nþ sinðtÞ and akðtÞ ¼ cosðtÞ=ðk� 1Þðk ¼ 2;3; . . . ;nÞ. The entry trajectories of matrix AðtÞ ¼ ½aijðtÞ� 2 Rn�n are shown inFig. 7 for the situation of n ¼ 5 and n ¼ 15. Evidently, matrix AðtÞ is strictly diagonally-dominant for any time instant t P 0,and is thus invertible [33].

Similarly, by applying the above-presented simulation techniques, ZNN model (1) can be exploited to invert the time-varying Toeplitz matrix AðtÞ. When design parameter c ¼ 1 and power-sigmoid activation functions (with n ¼ 4 andp ¼ 3) are used, the inversion performance of ZNN model (1) can be seen in Figs. 8 and 9. As shown in Fig. 8, starting fromfour randomly generated initial states within ½�1=ð2nÞ;1=ð2nÞ�, the state matrix XðtÞ ¼ ½xijðtÞ� 2 Rn�n (i; j ¼ 1;2; . . . ;n) of ZNNmodel (1) always converges to the theoretical time-varying inverse A�1ðtÞ (which corresponds to the convergence of residualerror kAðtÞXðtÞ � Ik to zero, as depicted in Fig. 9).

Example 3. In the above two illustrative examples, the time-varying matrices are inverted by using the presented ZNNmodel (1). In this example, the ZNN model and simulation techniques are exploited for the online inversion of a constant

0 1 2 3 4 5 6 7 8 9 100

5

1

5

2

5

3

γ = 1

time t (s)

0 1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

2.5

3

3.5

4

γ = 10

time t (s)

Fig. 6. Convergence of computational error kXðtÞ � A�1ðtÞk of perturbed ZNN (3).

0 2 4 6 8 10 12 14 16 18 203

4

5

6

7

0 2 4 6 8 10 12 14 16 18 202

1

0

1

2

aii (t)

a ij (t ) ( i = j ) a ij (t ) ( i = j )

time t (s)

time t (s)

(a) n = 5

0 2 4 6 8 10 12 14 16 18 2013

14

15

16

17

0 2 4 6 8 10 12 14 16 18 20−2

−1

0

1

2

aii(t)

time t (s)

time t (s)

(b) n = 15

Fig. 7. Entry trajectories of time-varying Toeplitz matrix AðtÞ ¼ ½aijðtÞ� 2 Rn�n .

0 2 4 6 8 10 12 14 16 18 20−0.1

−0.05

0

0.05

0.1

0.15

0.2

0.25

0.3xij (t)

time t (s) time t (s)

(a) n =50 2 4 6 8 10 12 14 16 18 20

−0.04

−0.02

0

0.02

0.04

0.06

0.08xij (t)

(b) n =15

Fig. 8. Online inversion of time-varying Toeplitz matrix AðtÞ by ZNN model (1).

0 2 4 6 8 10 12 14 16 18 200

0.5

1

1.5

2

2.5

3

3.5

4

n = 5

0 2 4 6 8 10 12 14 16 18 200

1

2

3

4

5

6

n = 15

time t (s)time t (s)

Fig. 9. Convergence of residual error kAðtÞXðtÞ � Ik by ZNN model (1) with c ¼ 1.

1612 Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617

matrix, which could be viewed as a special case of the present time-varying problem formulation and solution. The followingconstant matrix A 2 R3�3 [34] is presented, together with its theoretical inverse A�1 2 R3�3, to check the correctness of thesolutions generated by ZNN (1):

A ¼1 1 23 2 31 1 1

264

375; A�1 ¼

�1 1 �10 �1 31 0 �1

264

375:

By applying design parameter c ¼ 1 and power-sigmoid activation functions (with n ¼ 4 and p ¼ 3), the neural stateXðtÞ 2 R3�3 computed by the proposed ZNN model (1), starting from eight initial states randomly generated within[�2,2], could always converge to the theoretical inverse A�1. This is shown in Fig. 10, where dotted red lines correspondto the theoretical inverse A�1, and solid blue curves correspond to the neural-network solution XðtÞ.

In addition, solution error kXðtÞ � A�1k could be used to monitor the neural-network convergence. Fig. 11 illustrates thecharacteristics of the solution errors during the matrix-inversion process by using ZNN model (1), of which the neural stateXðtÞ starts from eight initial states randomly generated within [�2,2] as well. It can be seen from the figure that, by usingZNN model (1) to solve online for the inverse of the constant matrix A, the solution error is always convergent to zero.

In summary, the above theoretical and simulation results both substantiate the efficacy of the presented ZNN model (1)on time-varying and constant matrices inversion. In other words, the state matrix of ZNN model (1) can globally exponen-tially converge to the time-varying matrix inverse A�1ðtÞ (as well as the constant matrix inverse A�1).

0 1 2 3 4 5 6 7 8 9 10−3

−2

−1

0

1

2

3

4

5

x23(t)

x12 (t), x31 (t)

x21 (t), x32(t)

x11(t), x13(t), x22(t), x33(t)

time t (s)

Fig. 10. Online inversion of a constant matrix A 2 R3�3 by ZNN model (1).

0 1 2 3 4 5 6 7 8 9 100

1

2

3

4

5

6

7

γ = 1

time t (s)

Fig. 11. Convergence of solution error kXðtÞ � A�1k as of ZNN model (1).

Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617 1613

5. Conclusions

The general recurrent neural network proposed in [11] provides an effective online-computing approach for time-varyingmatrix inversion. This Zhang neural network (ZNN) was developed and generalized based on a matrix-valued error-function,instead of a scalar-valued norm-based energy-function usually associated with gradient-based (or Hopfield-type) neural net-works. This ZNN design method was able to make the solution error of a time-varying matrix inverse globally exponentiallyconverge to zero.

Differing from the conventional GNN models depicted in explicit dynamic equations, the proposed ZNN models, depictedin implicit dynamic equations, were able to make full use of the time-derivative information of time-varying matrices, andthus could effectively solve time-varying problems. Note that, the static problem can be viewed as a special case of the time-varying problem, and the time-derivative of the constant matrix equals zero. The ZNN efficacy can be seen in Figs. 1, 2, 9–11.In contrast, it follows from Figs. 3 and 4 that the GNN solution always lags behind the time-varying theoretical inverse with alarge solution error. This is possibly because the conventional GNN approach does not make use of the time-derivative infor-mation of the time-varying matrix AðtÞ. In addition, in view of the existence of nonlinearity and realization errors, by increas-ing the value of design parameter c and using better activation-function arrays (e.g., linear, sigmoid, or power-sigmoidactivation-function arrays), the solution error of the perturbed ZNN model can be greatly reduced. This point could be ob-served in Figs. 5 and 6.

Moreover, to simulate, code, and verify the resultant ZNN models, four important simulation techniques were worthy ofinvestigation in this paper, i.e., coding of activation-function arrays, Kronecker product of matrices, routine ‘‘ode45” with amass matrix, and the obtaining of matrix derivatives. Then, computer-simulation results, together with theoretical results,demonstrated the efficacy of the presented recurrent neural network model on time-varying (and constant) matrix inver-sion. Our future research directions may lie in the hardware implementation, robotic/control application, and experimental

1614 Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617

verification of the ZNN models. To date, the presented theoretical work has been applied to practical areas/applications, suchas the time-varying pseudoinverse resolution of redundant manipulators [11], the control of the inverted pendulum on a cartsystem [21], the design and implementation of computing-circuits [30], and the analysis of robots’ cyclic motion planning[35,36].

Acknowledgement

This work is supported by the Program for New Century Excellent Talents in University (NCET-07-0887). Since 1992, andbefore joining Sun Yat-Sen University (SYSU) in 2006, Yunong Zhang had been affiliated with the National University of Ire-land at Maynooth, the University of Strathclyde, the National University of Singapore, the Chinese University of Hong Kong,the South China University of Technology, and Huazhong University of Science and Technology. Yunong has been supportedby research fellowships, assistantship and studentships, which have been directly or indirectly related to this research. Yu-nong Zhang’s web-page is now at http://sist.sysu.edu.cn/%7Ezhynong/.

Appendix A. MATLAB codes related to Section 3

(I) Power-sigmoid activation-function array

function output=AFMpowersigmoid(E,xi,p)

if nargin==1, xi = 4; p = 3;elseif nargin == 2, p = 3;end

[noRows,noCols] = size(E);output=zeros(noRows,noCols);for i = 1:noRows

for j = 1:noCols

if abs(E(i,j))>=1output(i,j)=E(i,j) ^p;

else

outputði;jÞ ¼ ð1þ expð�xiÞÞ=ð1� expð�xiÞÞ � . . .

ð1� expð�xi � Eði;jÞÞÞ=ð1þ expð�xi � Eði;jÞÞÞ;end

endend

(II) User-defined function ‘‘ZnnRightHandSide”

function output=ZnnRightHandSide(t,x,gamma)

if nargin==2, gamma=1; end

n = sqrt(size(x,1));

% The following generates the time derivative of matrix M

syms u; D=diff(MatrixM(u));

u=t; dotM=eval(D);

% The following generates the vectorization of identity matrix I

vecI=reshape(eye(n),n ^2,1);% The following calculates the right hand side of Eq. (5)

% If using power-sigmoid function (firstly preferred):

output=-dotM*x-gamma*AFMpowersigmoid(MatrixM(t,x)*x-vecI);% If using sigmoid activation function (secondly preferred):

% output=-dotM*x-gamma*AFMsigmoid(MatrixM(t,x)*x-vecI);% If using linear activation function (thirdly preferred):

% output=-dot M*x-gamma*AFMlinear(MatrixM(t,x)*x-vecI);% If using power activation function (fourthly preferred):

% output=-dotM*x-gamma*AFMpower(MatrixM(t,x)*x-vecI);

(III) Mass matrix Mðt; xÞ :¼ I � AðtÞ as of Eq. (4)

function output=MatrixM(t,x)

% M(t,x) for ODE problem M(t,x)*x0=g(t,x)

A=[sin(t) cos(t);-cos(t) sin(t)];

output=kron(eye(size(A)),A);

Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617 1615

(IV) ODE solving with a state-independent mass matrix MðtÞ

tspan=[0 10]; x0=4*(rand(4,1)-0.5*ones(4,1));options=odeset(’Mass’,@MatrixM,’MStateDep’,’none’);

[t,x]=ode45(@ZnnRightHandSide,tspan,x0,options);

Appendix B. MATLAB codes related to Section 4

(I) The code calling the user-defined function ‘‘ZnnRightHandSide”

gamma=1; %just for illustrative purposes

tspan=[0 10]; options=odeset(’Mass’,@MatrixM,’MStateDep’,’none’);

for iter=1:8

x0=4*(rand(4,1)-0.5*ones(4,1));[t,x]=ode45(@ZnnRightHandSide,tspan,x0,options,gamma);

subplot(2,2,1);plot(t,x(:,1));hold on

subplot(2,2,3);plot(t,x(:,2));hold on

subplot(2,2,2);plot(t,x(:,3));hold on

subplot(2,2,4);plot(t,x(:,4));hold on

end

subplot(2,2,1);plot(t,sin(t),‘r:’); %compare with A�1ðtÞsubplot(2,2,3);plot(t,cos(t),‘r:’);

subplot(2,2,2);plot(t,-cos(t),‘r:’);

subplot(2,2,4);plot(t,sin(t),‘r:’);

(II) User-defined functions ‘‘NormError” and ‘‘ZnnNormError”

function NormError(x0,gamma)

tspan=[0 10];

options=odeset(’Mass’,@MatrixM,’MStateDep’,’none’);

[t,x]=ode45(@ZnnRightHandSide,tspan,x0,options,gamma);

Ainv=[sin(t) cos(t) -cos(t) sin(t)]’;

err=x’-Ainv; total=length(t);

for i=1:total,

nerr(i)=norm(err(:,i));

end

plot(t,nerr); hold on

function ZnnNormError(gamma)

% usage: figure; ZnnNormError(1);

if nargin<1, gamma = 1; end

for i=1:8

x0=4*(rand(4,1)-0.5*ones(4,1));

NormError(x0,gamma);

end

(III) User-defined function ‘‘ZnnRightHandSideImprecise”

function output=ZnnRightHandSideImprecise(t,x,gamma)

if nargin==2, gamma=1; end

e2=.5; e3=.5;

deltaB=e2*[cos(3*t) -sin(3*t); sin(3*t) cos(3*t)];

kronB=kron(eye(2), deltaB);

deltaC=e3*[sin(3*t) 0; 0 cos(3*t)];

vecC=reshape(deltaC,4,1);

vecI=reshape(eye(2),4,1);

syms u; D=diff(MatrixM(u)); u=t; dotM=eval(D);

output=-(dotM+kronB)*x-gamma*. . .

AFMpowersigmoid(MatrixM(t)*x-vecI)+vecC;

1616 Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617

(IV) User-defined function ‘‘ZnnRobust”

function ZnnRobust(gamma)

tspan=[0 10];

options=odeset(’Mass’,@MatrixM,’MStateDep’,’none’);

for iter=1:8

x0=4*(rand(4,1)-0.5*ones(4,1));

[t,x]=ode45(@ZnnRightHandSideImprecise,tspan,x0,options,gamma);

subplot(2,2,1);plot(t,x(:,1));hold on

subplot(2,2,3);plot(t,x(:,2));hold on

subplot(2,2,2);plot(t,x(:,3));hold on

subplot(2,2,4);plot(t,x(:,4));hold on

end

subplot(2,2,1);plot(t,sin(t),’r:’);

subplot(2,2,3);plot(t,cos(t),’r:’);

subplot(2,2,2);plot(t,-cos(t),’r:’);

subplot(2,2,4);plot(t,sin(t),’r:’);

References

[1] Y. Zhang, Towards piecewise-linear primal neural networks for optimization and redundant robotics, in: Proceedings of IEEE International Conferenceon Networking, Sensing and Control, 2006, pp. 374–379.

[2] Y. Zhang, J. Wang, A dual neural network for convex quadratic programming subject to linear equality and inequality constraints, Physics Letters A 298(4) (2002) 271–278.

[3] R.J. Steriti, M.A. Fiddy, Regularized image reconstruction using SVD and a neural network method for matrix inversion, IEEE Transactions on SignalProcessing 41 (10) (1993) 3074–3077.

[4] Y. Zhang, W.E. Leithead, D.J. Leith, Time-series Gaussian process regression based on Toeplitz computation of OðN2Þ operations and OðNÞ-level storage,in: Proceedings of the 44th IEEE Conference on Decision and Control, 2005, pp. 3711–3716.

[5] T. Sarkar, K. Siarkiewicz, R. Stratton, Survey of numerical methods for solution of large systems of linear equations for electromagnetic field problems,IEEE Transactions on Antennas and Propagation 29 (6) (1981) 847–856.

[6] Y. Zhang, Revisit the analog computer and gradient-based neural system for matrix inversion, in: Proceedings of IEEE International Symposium onIntelligent Control, 2005, pp. 1411–1416.

[7] R.H. Sturges Jr., Analog matrix inversion (robot kinematics), IEEE Journal of Robotics and Automation 4 (2) (1988) 157–162.[8] W.E. Leithead, Y. Zhang, OðN2Þ-operation approximation of covariance matrix inverse in Gaussian process regression based on quasi-Newton BFGS

method, Communications in Statistics-Simulation and Computation 36 (2) (2007) 367–380.[9] S. Van Huffel, J. Vandewalle, Analysis and properties of the generalized total least squares problem AX B when some or all columns in A are subject to

errors, SIAM Journal on Matrix Analysis and Applications 10 (1989) 294–315.[10] Y. Zhang, S.S. Ge, A general recurrent neural network model for time-varying matrix inversion, in: Proceedings of the 42nd IEEE Conference on Decision

and Control, 2003, pp. 6169–6174.[11] Y. Zhang, S.S. Ge, Design and analysis of a general recurrent neural network model for time-varying matrix inversion, IEEE Transactions on Neural

Networks 16 (6) (2005) 1477–1490.[12] Y. Zhang, Z. Chen, K. Chen, B. Cai, Zhang neural network without using time-derivative information for constant and time-varying matrix inversion, in:

Proceedings of International Joint Conference on Neural Networks, 2008, pp. 142–146.[13] Y. Zhang, W. Ma, C. Yi, The Link between Newton iteration for matrix inversion and Zhang neural network (ZNN), in: Proceedings of IEEE International

Conference on Industrial Technology, 2008, pp. 1–6.[14] J. Jang, S. Lee, S. Shin, An optimization network for matrix inversion, in: D.Z. Anderson (Ed.), Neural Information Processing Systems, American Institute

of Physics, New York, 1988, pp. 397–401.[15] J. Wang, A recurrent neural network for real-time matrix inversion, Applied Mathematics and Computation 55 (1993) 89–100.[16] R.K. Manherz, B.W. Jordan, S.L. Hakimi, Analog methods for computation of the generalized inverse, IEEE Transactions on Automatic Control 13 (5)

(1968) 582–585.[17] K.S. Yeung, F. Kumbi, Symbolic matrix inversion with application to electronic circuits, IEEE Transactions on Circuits and Systems 35 (2) (1988) 235–

238.[18] A. El-Amawy, A systolic architecture for fast dense matrix inversion, IEEE Transactions on Computers 38 (3) (1989) 449–455.[19] Y.Q. Wang, H.B. Gooi, New ordering methods for sparse matrix inversion via diagonalization, IEEE Transactions on Power Systems 12 (3) (1997) 1298–

1305.[20] N.C.F. Carneiro, L.P. Caloba, A new algorithm for analog matrix inversion, in: Proceedings of the 38th Midwest Symposium on Circuits and Systems,

1995, pp. 401–404.[21] Y. Zhang, D. Jiang, J. Wang, A recurrent neural network for solving Sylvester equation with time-varying coefficients, IEEE Transactions on Neural

Networks 13 (5) (2002) 1053–1063.[22] Y. Zhang, S. Yue, K. Chen, C. Yi, MATLAB simulation and comparison of Zhang neural network and gradient neural network for time-varying Lyapunov

equation solving, in: LNCS Proceedings of the Fifth International Symposium on Neural Networks, vol. 5263, 2008, pp. 117–127.[23] Y. Zhang, H. Peng, Zhang neural network for linear time-varying equation solving and its robotic application, in: Proceedings of the Sixth International

Conference on Machine Learning and Cybernetics, 2007, pp. 3543–3548.[24] Y. Zhang, Z. Li, C. Yi, K. Chen, Zhang neural network versus gradient neural network for online time-varying quadratic function minimization, in: LNAI

Proceedings of International Conference on Intelligent Computing, vol. 5227, 2008, pp. 807–814.[25] Y. Zhang, C. Yi, W. Ma, Comparison on gradient-based neural dynamics and Zhang neural dynamics for online solution of nonlinear equations, in: LNCS

Proceedings of International Symposium on Intelligence Computation and Applications, vol. 5370, 2008, pp. 269–279.[26] F. Fourati, M. Chtourou, A greenhouse control with feed-forward and recurrent neural networks, Simulation Modelling Practice and Theory 15 (2007)

1016–1028.[27] K. Gulez, R. Guclu, CBA-neural network control of a non-linear full vehicle model, Simulation Modelling Practice and Theory 16 (2008) 1163–1176.[28] L. Ekonomou, G.P. Fotis, T.I. Maris, P. Liatsis, Estimation of the electromagnetic field radiating by electrostatic discharges using artificial neural

networks, Simulation Modelling Practice and Theory 15 (2007) 1089–1102.[29] C. Mead, Analog VLSI and Neural Systems, Addison-Wesley, Reading, MA, 1989.

Y. Zhang et al. / Simulation Modelling Practice and Theory 17 (2009) 1603–1617 1617

[30] C. Yi, Y. Zhang, Analogue recurrent neural network for linear algebraic equation solving, Electronics Letters 44 (18) (2008) 1078–1079.[31] The MathWorks, Inc., Matlab 7.0, Natick, MA, 2004.[32] R.A. Horn, C.R. Johnson, Topics in Matrix Analysis, Cambridge University Press, Cambridge, 1991.[33] J. Weickert, B.M.H. Romeny, M.A. Viergever, Efficient and reliable schemes for nonlinear diffusion filtering, IEEE Transactions on Image Processing 7 (3)

(1998) 398–410.[34] F.M. Ham, I. Kostanic, Principles of Neurocomputing for Science and Engineering, McGraw-Hill, New York, 2001.[35] K. Chen, L. Zhang, Y. Zhang, Cyclic motion generation of multi-link planar robot performing square end-effector trajectory analyzed via gradient-

descent and Zhang et al’s neural-dynamic methods, in: Proceedings of the Second International Symposium on Systems and Control in Aeronautics andAstronautics, 2008, pp. 1–6.

[36] Y. Zhang, X. Lv, Z. Yang, Z. Li, Angle-drift elimination scheme and its analysis for redundant robot manipulators, Microcomputer Information 25 (1)(2009) 265–267 (in Chinese).