Zeroing Neural Networks, an Introduction to, a Partial ...

Zeroing Neural Networks,an Introduction to, a Partial Survey of, andPredictive Computations for Discretized

Time-varying Matrix ProblemsFrank Uhlig *

AbstractThis paper is designed to increase the knowledge and computational know-how for time-varying matrixproblems and Zeroing Neural Networks in the West. Zeroing Neural Networks (ZNN) were invented fortime-varying matrix problems 20 years ago in China and almost all of their advances have been made inand still come from its birthplace. ZNN methods have become a backbone for solving discretized sen-sor driven time-varying matrix problems in real-time, in theory and in on-chip applications for robots, incontrol theory and in engineering. They have become the method of choice for many time-varying matrixproblems that benefit from or require efficient, accurate and predictive real-time computations. The typicaldiscretized ZNN algorithm needs seven distinct steps for its initial set-up. The construction of discretizedZNN algorithms starts from a model with its associated error equation and the stipulation that the error func-tion decrease exponentially fast. The error function differential equation is then mated with a convergentlook-ahead finite difference formula to create a distinctly new multistep-style solver that predicts the futurestate of the system reliably from current and earlier state and solutions data. Matlab codes for discretizedZNN algorithms typically consist of one linear equations solve and one recursion of already available dataper time step. This makes discretized ZNN based algorithms highly competitive with ordinary differentialequation initial value path following or homotopy methods that are designed to work adaptively and givesZNN different characteristics and applicabilities from multistep ODE initial value solvers. DiscretizedZNN methods can solve problems given by sensor data with constant sampling gaps or from functionalequations. To illustrate the easy adaptability of discretized ZNN and further the understanding of ZNN, thispaper details the seven step set-up of ZNN for 11 separate time-varying problems and supplies new codesfor seven. Open problems are mentioned as well as detailed references to recent work on discretized ZNNand time-varying matrix problems and comparisons with our standard multistep methods for ODEs.

Keywords: time-varying matrix problem, neural network, zeroing neural network, discretized ZNN algorithm,matrix flow, time-varying numerical algorithm, predictive numerical method

AMS : 65-02, 65-04, 65F99, 65F30, 15-04, 15A99, 15B99

1 IntroductionThis is an introduction and overview of a small part of a seemingly new area of Numerical Linear Algebra anda new class of multistep formulas. The paper deals with time-varying matrices and therein with how to solvestandard numerical problems of matrix analysis when the matrices are parameter dependent and their data eitherarrives in discretized form from clocked sensors or is derived at equidistant discrete times from formulaic matrixmodels. Time-varying or parameter-varying matrices will be called matrix flows. In this paper we only studydiscretized Zeroing Neural Networks or Zhang Neural Networks (ZNN). Continuous time-varying matrix flowproblems can be easily and accurately solved by finely discretized Zeroing Neural Networks. We will not attemptto study all known variations of ZNN here due to the overwhelming wealth of applications and specializations thathave evolved over time with nearly 500 papers and a handful of books. ZNN has never been studied by numericalanalysts. Yet it poses many open problems. But its workings have never been explained for the ZNN methods thatare ubiquitously used in modern engineering applications, mainly in the East.

*Department of Mathematics and Statistics, Auburn University, Auburn, AL 36849-5310 ([email protected])

1

arX

iv:2

008.

0272

4v3

[m

ath.

NA

] 2

1 O

ct 2

021

Discretized ZNN and Time-varying Matrix Flows 2

The discretized ZNN method represents a special class of Recurrent Neural Networks that originated some 40years ago and that were intended to solve dynamical systems. The ZNN method was proposed twenty years ago byYunong Zhang and Jun Wang in 2001, see [38]. Zhang and Wang’s approach started from a global error functionand an error differential equation to achieve exponential error decay for the solutions of time-varying matrix andvector systems. At the time Yunong Zhang was a Ph D candidate at Chinese University in Hong Kong and JunWang his advisor. Since then, Zhang Neural Networks have become one mainstay for predictive time-varyingmatrix flow computations in the engineering world. Discretized ZNN methods help nowadays with optimizing andcontrolling robot behavior, with autonomous vehicles, chemical plant control, et cetera. They are extremely swiftand accurate in their predictive numerical matrix flow computations.A Caveat : The term Neural Network has many uses.

Its origin lies in biology and medicine. There it refers to the neuron network of the brain and to the synapsesof a nervous system. In applied mathematics neural networks are generally associated with models and prob-lems that mimic or follow a brain-like function or use nervous system like algorithms that pass informationalong.In the computational sciences of today, assignations that use terms like neural network most often refer tonumerical algorithms that search for relationships in parameter-dependent data or that deal with time-varyingproblems. The earliest numerical use of neural networks stem from the late 1800s. Today’s ever evolvingnumerical ‘neural network’ methods may involve deep learning or large data and data mining. They occur inartificial neural networks, with recurrent neural networks (RNN), with continuation methods for differentialequations, in homotopy methods and in the numerical analysis of dynamical systems, as well as in artificialintelligence (AI), in machine learning, and in image recognition and restoration, and so forth. In each of theserealizations of ’neural network’ ideas, different algorithms are generally used for differing problems.Zhang neural networks (ZNN) differ from all of the above and are specialized to solve time-varying matrixand matrix based optimization problems. Unfortunately, both time-varying matrix problems and continu-ous and discretized ZNN methods to solve them, have no home in the 2010 or 2020 Mathematics SubjectClassifications and they are not listed or mentioned in Wikipedia at all.

In practice, ZNN methods and time-varying matrix flow problems are clearly governed by different mathematicalprinciples and are subject to different quandaries than those of static matrix analysis where Wilkinson’s backwardstability and error analysis are common and beautiful modern principles reign. In fact ZNN methods cannot solvestatic matrix problems at all. Note that discretized ZNN processes are predictive by design and therefore requirelook-ahead convergent finite difference schemes that have only rarely occurred or been used before, such as in thecorrector phase of multistep ODE solvers. Time-varying matrix computations form a new, separate and still mainlyuncharted territory of Numerical Linear Algebra that is well worth studying, coding and learning about.

This paper is divided into two parts: Section 2 will explain the discretized ZNN set-up process in detail, see Re-mark 1. In Section 3 we list and exemplify a number of models and applied problems for time-varying matrixphenomena that engineers are now solving or beginning to solve via discretized ZNN matrix algorithms.

2 The Workings of Discretized ZNN Methods for Parameter-varying Ma-trix Flows

For simplicity and by the limits of space for two decades of ZNN based engineering research and use, we re-strict our attention to time-varying matrix problems in discretized form throughout this paper. ZNN methods workequally well with continuous matrix inputs and continuous ZNN. It has often been tested for consistency, conver-gence, stability and its behavior in the presence of noise, see some of the References or search the vast literaturein Google.For discretized time-varying data and any matrix problem therewith, all discretized Zeroing Neural Network meth-ods proceed using the following identical seven construction steps – after appropriate start-up value preparations.


Suppose for a moment that we are given a continuous time-varying matrix/vector model

F (A(t), B(t), x(t), ..) = g(t, C(t), u(t), ..) (1)

with a time-varying unknown vector or matrix x(t) and with compatibly sized time-varying matrices A(t), B(t),C(t), ... and time-varying vectors u(t), .. that are known at discrete and equidistant time instances ti for i ≤ kand k = 1, ... such as from sensor data. Steadily timed sensor data is ideal for discretized ZNN. Our task withdiscretized ZNN is to find the solution x(tk+1) of (1) accurately and in real-time from earlier x(t..) values andearlier matrix and vector data. Note that here the ’unknown’ x(t) might be a concatenated vector or an augmentedmatrix x(t) that - for example - may contain both, the eigenvector matrix and the associated eigenvalues for atime-varying matrix eigenvalue problem. Then the given flow matrices A(t) and others might have to be enlargedsimilarly to stay compatible with the added eigendata vector x(t) and possibly also for any other vectors or matricesB(t), u(t), and so forth.

Step 1 : From a given model equation (1), form the error function

E(t) = F (A(t), B(t), x(t), ..)− g(t, C(t), u(t), ..) (2)

which should ideally be zero, i.e., E(t) = 0 for all t ∈ [to, tf ] if x(t) actually solves (1) in the desiredinterval.

Step 2 : Take the derivative E(t) of the error function E(t) and stipulate its exponential decay:Demand that

E(t) = −η E(t) (3)

for some fixed constant η > 0 in case of Zhang Neural Networks (ZNN).Or demand that

E(t) = −γ F(E(t))

for γ > 0 and a monotonically nonlinear increasing activation function F . That would change E(t)element-wise and give us a different method, called a Recurrent Neural Network (RNN).

The right-hand sides for ZNN and RNN methods differ subtly. Exponential error decay and thus convergence to theexact solution x(t) of (1) is automatic for both variants. Depending on the problem, different activation functionsF are used in the RNN version such as linear, power sigmoid or hyperbolic sine functions. These can result indifferent and better problem suited convergence properties with RNN, see [39], [8], or [35] e.g. for examples.The error exponential decay stipulation (3) is much more stringent than a PID controller would be as it does bringthe entry-wise solution errors down to truncation error levels exceedingly fast after start-up and from any startingvalue.In this paper we will, however, limit our attention to discretized Zhang Neural Networks (ZNN) exclusively fromnow on for simplicity.Step 3 : Solve the exponentially decaying error equation differential equation (3) of Step 2 algebraically for

x(tk) = . . . if possible. If impossible, reconsider the problem, revise the model, and try again.(This behavior will be encountered and mitigated here and in Section 2.)

Step 4 : Select a look-ahead convergent finite difference formula for the desired truncation error order O(τ j+2)that expresses x(tk) in terms of x(tk+1), x(tk), ..., x(tk−(j+s)+2) in terms of j + s known data pointsusing the table of known convergent look-ahead finite difference formulas of type j s in [23] and [24].

Step 5 : Equate the x(tk) derivative terms in Steps 3 and 4 and thereby dispose of x(tk) or the ODE problemaltogether.

Step 6 : Solve the solution-derivative free linear equation obtained in Step 5 for x(tk+1) and iterate.Step 7 : Increase k+ 1 to k+ 2 and up all data of Step 6; then solve the updated recursion for x(tk+2). And repeat

until t... ≥ tf .

Most recently, steps 3 - 5 above have been streamlined in [17] by equating the Adams-Bashforth difference for-mula, see e.g. [1, p. 458-460], applied to the derivatives x(t...) with a convergent look-ahead difference formula


from [23] for x(tk). To achieve overall convergence, this new process requires a delicate balance between the pairof finite difference formulas; for further details see [17].

Discretized ZNN methods are highly accurate and quickly converging due to the stipulated exponential error decayof Step 2. They are still evolving and beckon numerical analysis scrutiny and explanations.

The errors of ZNN methods have two sources: for one, the chosen finite difference formula’s truncation error orderin Step 4 depends on the sampling gap τ = tk+1 − tk = const, and secondly on the conditioning and roundingerrors of the linear equation solves in Step 6. Discretized ZNN methods are designed to give us the future solutionvalue x(tk+1) accurately and they do so immediately after time tk from current and earlier equidistant sensor andsolution data.A convergent look-ahead finite difference scheme has only been used once in the Adams-Moulton multistep for-mula and we know of no other occurrences in the literature prior to ZNN. Discrete ZNN methods generally reducetime-varying matrix problems of the ’starting kind’ (1) to a recurrence relation and a linear equations solve withsmall computational costs per time step. These methods can be easily transferred to on-board chip designs fordriving and controlling robots. See [41] for 13 separate time-varying matrix/vector tasks, their Simulink modelsand circuit diagrams, as well as two chapters on fixed-base and mobile robot applications. Each chapter in [41] isreferenced with 10 to 30 plus citations from the engineering literature.Zeroing Neural Networks have been used extensively in engineering and design for two decades now, but thenumerical analysis of ZNN has hardly been started. Time-varying matrix numerical analysis seems to be very dif-ferent from static matrix analysis. It seems to depend on and run according to different principles than Wilkinson’snow classic backward stability and error analysis based static matrix methods. This will become clear and clearerthroughout this introductory survey paper.

Readers probably might have had little need for numerical methods that predict future system states in real timefor time-varying matrix and vector problems. We start by exemplifying the time-varying eigenvalue problemA(t)x(t) = λ(t)x(t) with hermitean matrix flows A(t) = A(t)∗ ∈ Cn,n and we lead through the seven stepspath of discretized ZNN. This is followed by an analysis of the fundamental difference between discretized ZNNmethods for time-varying matrix problems and one-step and multi-step ODE path followers.If An,n = A∗ is a fixed static matrix, one would likely solve the eigenvalue problem via Francis multi-shiftimplicit QR algorithm if n ≤ 11, 000 and use Krylov methods for larger sized A. Eigenvalues are continuousfunctions of their matrix entries. Thus taking the computed eigenvalues for A(tk) as an approximation for theeigenvalues of A(tk+1) might seem to suffice if the sampling gap τ = tk+1− tk is relatively small. But in practicethe eigenvalues of A(tk) often share only a few correct leading digits with the eigenvalues of A(tk+1). In factthe difference between any pair of respective eigenvalues of A(tk) and A(tk+1) are generally of size O(τ), see[9, 36, 37]. Hence there is need for different methods that deal predictively accurate with time-varying matrixflows A(t).

By the definition of eigenvalues, for a given hermitean matrix flow A(t) = A(t)∗ ∈ Cn,n (or any diagonalizablematrix flow A(t)) we need to compute a nonsingular matrix flow V (t) ∈ Cn,n and a diagonal time-varying matrixflow D(t) ∈ Cn,n so that

A(t)V (t) = V (t)D(t) for all t ∈ [t0, tf ]. (1∗)

This is our model for the time-varying eigenvalue problem.Here are the steps for time-varying matrix eigen-analyses with ZNN.Step 1 : Create the error function

E(t) = A(t)V (t)− V (t)D(t) (= On,n ideally.) (2∗)

Step 2 : Stipulate exponential decay of E(t) as a function of time, i.e.,

E(t) = −η E(t) (3∗)


for a decay constant η > 0. (RNN methods would use different entry decaying functions on the right-handside.)Equation (3∗), written out explicitly is

E(t) = A(t)V (t) +A(t)V (t)− V (t)D(t)− V (t)D(t)

(∗)= −ηA(t)V (t) + ηV (t)D(t) = −ηE(t), . (4∗)

Rearranged with all derivatives of the unknowns V (t) and D(t) gathered on the left-hand side of (∗) :

A(t)V (t)− V (t)D(t)− V (t)D(t) = −ηA(t)V (t) + ηV (t)D(t)− A(t)V (t) . (5∗)

Unfortunately we do not know how to solve the full system eigen-equation (4∗) algebraically for the eigendataderivative matrices V (t) and D(t) by using simple matrix algebra as Step 3 asks us to do. This is due to the non-commutativity of matrix products and since the unknown derivative V (t) appears both as a left and right matrixfactor in (4∗). A solution that relies on Kronecker products for symmetric matrix flows A(t) = A(t)T is availablein [43] and we will go the Kronecker product route later on when dealing with square roots of time-varying matrixflows in subparts (VII) and (VII start-up) and also when solving time-varying classic matrix equations in subpart(IX) of Section 3.

Now we want to revise our matrix eigendata model and restart the whole process anew. To overcome the abovedilemma, we separate the time-varying matrix global eigenvalue problem for An,n(t) into n eigenvalue problems

A(t)xi(t) = λi(t)xi(t) and i = 1, ..., n (1i)

that can be solved for one eigenvector and one eigenvalue at a time as follows.Step 1 : Here the error function is

e(t) = A(t)xi(t)−λi(t)xi(t) (= On,1 ideally.) (2i)

Step 2 : Demand exponential decay of e(t) as a function of time, i.e.,

e(t) = −η e(t) (3i)

for a decay constant η > 0.Equation (3i), written out becomes

e(t) = A(t)xi(t) +A(t)xi(t)− λi(t)xi(t)− λi(t)xi(t)(∗)= −η A(t)xi(t) + η λi(t)xi(t) = −η e(t), (4i)

or rearranged with the derivatives of xi(t) and λi(t) gathered on the left-hand side of (∗) in (4i) :

A(t)xi(t)−λi(t)xi(t)−λi(t)xi(t) = −η A(t)xi(t)+η λi(t)xi(t)−A(t)xi(t) . (5i)

Combining the xi derivative terms gives us

(A(t)−λi(t)In)xi(t)−λi(t)xi(t) = (−η (A(t)−λi(t)In)−A(t)) xi(t) . (6i)

For each i = 1, ..., n equation (6i) is a differential equation in the unknown eigenvector xi(t) ∈ Cn and the

unknown eigenvalue λi(t) ∈ C. We concatenate xi(t) and λi(t) in zi(t) =

(xi(t)λi(t)

)∈ Cn+1 and obtain the

following system of DEs for the unknown eigenvector xi(t) and its associated eigenvalue λi(t)(A(t)− λi(t)In −xi(t)

)n,n+1

(xi(t)

λi(t)

)=(−η (A(t)− λi(t)In)− A(t)

)xi(t) (7i)


where the augmented system matrix on the left hand side of (7i) has dimensions n by n+ 1 if A is n by n. As eacheigenvector of a matrix defines an invariant 1-dimensional subspace we must insure that the computed eigenvectorsxi(t) of A(t) do not grow infinitely small or infinitely large. Thus we require the computed eigenvectors toattain unit length asymptotically by introducing the additional error function e2(t) = x∗i (t)xi(t) − 1. Stipulatingexponential decay for e2 leads to

e2(t) = 2x∗i (t)xi(t) = −µ (x∗i (t)xi(t)− 1) = −µ e2(t)

or−x∗i (t)xi(t) = µ/2 (x∗i (t)xi(t)− 1) (8i)

for a second decay constant µ > 0. If we set µ = 2η, place equation (8i) below the last row of the n by n + 1system matrix of equation (7i), and extend its right hand side vector by the right hand side entry in (8i), we obtainan n+ 1 by n+ 1 time-varying system of DEs with a hermitean system matrix if A(t) is hermitean. I.e.,(

A(t)− λi(t)In −xi(t)−x∗i (t) 0

)(xi(t)

λi(t)

)=

((−η (A(t)− λi(t)In)− A(t))xi(t)

η (x∗i (t)xi(t)− 1)

). (9i)

We set

P (tk) =

(A(tk)− λi(tk)In −xi(tk)−x∗i (tk) 0

)∈ Cn+1,n+1, z(tk) =

(xi(tk)λi(tk)

)∈ Cn+1 ,

and q(tk) =

((−η (A(tk)− λi(tk)In)− A(tk))xi(tk)

η (x∗i (tk)xi(tk)− 1)

)∈ Cn+1

for discretized times t = tk. If A(t) is hermitean, so is each P (tk) and we have completed Step 3:Step 3 : Our model (1i) for the ith eigenvalue equation has been transformed into the matrix/vector differential

equation

P (tk)z(tk) = q(tk) or z(tk) = P (tk)\q(tk) . (10i)

Step 4 : We choose the following convergent look-ahead finite 5-IFD (five Instance Finite Difference) formula oftype j s = 2 3 with truncation error order O(τ3) from [24] for zk:

zk =8zk+1 + zk − 6zk−1 − 5zk−2 + 2zk−3

18τ∈ Cn+1 . (11i)

Step 5 : Equating the different expressions for 18τ zk in (11i) and (10i) we have

18τ ·zk = 8zk+1+zk−6zk−1−5zk−2+2zk−3(∗)= 18τ ·(P\q) = 18τ ·zk (12i)

with error order O(τ4) that has increased by 1 due to the multiplication of (10i) and (11i) by 18τ .

Step 6 : Here we solve equation (∗) in (12i) for zk+1 to obtain the discretized ZNN recursion

zk+1 =9

4τ(P (tk)\q(tk))−1

8zk+

3

4zk−1+

5

8zk−2−

1

4zk−3 ∈ Cn+1 (13i)

with truncation error order O(τ4).

Step 7 : Iterate to predict the eigendata zk+2 for A(tk+2) from earlier eigen and system data for tjj withjj ≤ k + 1.

The final formula (13i) contains the computation formula for future eigendata with near unit eigenvectors in thetop n entries of z(tk+1) via ZNN. Only this formula needs to be implemented in code. All of its entries have beencomputed earlier as eigendata for times tjj with jj ≤ k, except for the system or sensor input A(tk) and A(tk),the later of which is best computed via a high error order derivative formula from previous A(tjj) with jj ≤ k.


The only computational the code lines for performing the ZNN iteration step for one of the eigenvalues from timet = tk to tk+1 are below. There ze denotes one eigenvalue of Al = A(tk) and zs its associated eigenvector. Zjis the relevant set of earlier eigendata for A(tj) with j ≤ k and Adot an approximation of A(tk). Finally eta, tauand taucoeff are chosen for the desired error order finite difference formula and its characteristic polynomial,respectively.

. . .

. . .Al(logicIn) = diag(Al) - ze*ones(n,1); % Al = A(tk) - ze*InP = [Al,-zs;-zs’,0]; % P is generally not hermiteanq = taucoeff*tau*[(eta*Al + Adot)*zs; -1.5*eta*(zs’*zs-1)]; % rh sideX = linsolve(P,q); % solve a linear equationZnew = -(X + Zj*polyrest); % New eigendata at t_{k+1}ZN(:,jj) = Znew; % Extend the known eigendata

. . .

. . .

The four barred central codes lines above express the computational essence of Step 6 above for the matrix eigen-problem in ZNN. Only there is any math performed, the rest of the program code are input reads and output savesand preparations. After Step 6 we store the new data and repeat these 4 code lines with peripherals for tk+2 untilwe are done with one eigenvalue. Then we repeat the same code for the next eigenvalue of a hermitean or diago-nalizable matrix flow A(t).

The 5-IFD formula (11i) is of type j s = 2 3 and when used in discretized ZNN its truncation error order be-comes O(τ4) as j + 2 = 2 + 2 = 4. To start a discretized ZNN iteration process with a look-ahead convergentfinite difference formula of type j s from the list in [24] requires j + s known starting values. For time-varyingmatrix eigenvalue problems that are given by function inputs for A(tk) we generally use Francis QR to generatej + s initial eigendata sets, then iterate via discretized ZNN and throughout the iterative process we need to keeponly the most recent j + s data sets in memory to update to the next time’s tk+1 solution.MATLAB codes for time-varying matrix eigenvalue computations via discretized ZNN are available at [26].

How can we construct general look-ahead and convergent finite difference schemes of arbitrary truncation errororders O(τp) for use in discretized ZNN methods? We shall start from random entry seed vectors and use Taylorpolynomials and elementary linear algebra that are look-ahead but may or – most likely – may not be convergent.The constructive first step of finding look-ahead finite difference formulas is followed by a second optimizationprocedure to find look-ahead and convergent finite difference formulas of the desired error order. This secondnon-linear part may not succeed as we shall explain further below.Consider a discrete time-varying state vector xk = x(tk) = x(k · τ) for a constant sampling gap τ and k =0, 1, 2, ... and write out m = ` + 1 explicit Taylor expansions for xk+1, xk−1, ..., xk−` about xk as follows forpredictive ZNN.Each Taylor expansion in our scheme will contain j + 3 terms on the right hand side, namely j + 1 derivativeterms, a term for xk, and one for the error O(τ j+2). Each right hand side’s under- and over-braced j ’column’terms below contain products of identical powers of τ and identical partial derivatives of xk which - combined incolumn vector form, we will call taudx. Our aim is to find a linear combination of these m = ` + 1 equationsfor which the under- and over-braced sums vanish for all possible higher derivatives of the solution x(t). If we areable to do so, then we can express xk+1 in terms of xl for l = k, k − 1, ..., k − `, xk, and τ with a truncation errorof order O(τ j+2) by using the found linear combination for a shortened version of the m = ` + 1 equations (14)through (19) below.


j terms

xk+1 = xk + τ xk

︷︸︸︷+

τ2

2!xk +

τ3

3!

...xk ... +

τ j+1

(j + 1)!

j+1

xk + O(τ j+2) (14)

xk−1 = xk − τ xk +τ2

2!xk − τ3

3!

...xk ... + (−1)j+1 τ j+1

(j + 1)!

j+1

xk + O(τ j+2) (15)

xk−2 = xk − 2τ xk +(2τ)2

2!xk −

(2τ)3

3!

...xk ... + (−1)j+1 (2τ)j+1

(j + 1)!

j+1

xk + O(τ j+2) (16)

xk−3 = xk − 3τ xk +(3τ)2

2!xk −

(3τ)3

3!

...xk ... + (−1)j+1 (3τ)j+1

(j + 1)!

j+1

xk + O(τ j+2) (17)

...... (18)

xk−` = xk − `τ xk +(`τ)2

2!xk −

(`τ)3

3!

...xk ... + (−1)j+1 (`τ)j+1

(j + 1)!

j+1

x k︸︷︷︸ + O (τ j+2) (19)

j termsWe first isolate the ’rational number’ factors in the ’braced j columns’ on the right hand side of equations (14)through (19) in the rational `+ 1 by j matrix

A`+1,j =

1

2!

1

3!

1

4!· · · · · · 1

(j + 1)!

1

2!− 1

3!

1

4!· · · · · · (−1)j+1 1

(j + 1)!

22

2!−23

3!

24

4!· · · · · · (−1)j+1 2j+1

(j + 1)!...

......

...

`2

2!−`

3

3!

`4

4!· · · · · · (−1)j+1 `j+1

(j + 1)!

. (20)

By design, A has `+ 1 rows and j columns. The over- and underbraced expressions in equations (14) through (19)have the matrix times vector product form

A · taudx =

1

2!

1

3!

1

4!· · · · · · 1

(j + 1)!

1

2!− 1

3!

1

4!· · · · · · (−1)j+1 1

(j + 1)!

22

2!−23

3!

24

4!· · · · · · (−1)j+1 2j+1

(j + 1)!...

......

...

`2

2!−`

3

3!

`4

4!· · · · · · (−1)j+1 `j+1

(j + 1)!

`+1,j

τ2 xk

τ3...xk

τ44xk......

τ j+1j+1

xk

j,1

. (21)

Here the j-dimensional column vector taudx contains the increasing powers of τ multiplied by the respectivehigher derivatives of xk that were pulled out from equations (14) to (19) when forming the matrix A with itsrational entries.


Note that for any nonzero left kernel row vector y ∈ R`+1 of A`+1,j with y ·A = O1,j we have

y1,`+1 ·A`+1,j · taudxj,1 = O1,j · taudxj,1 = 0 ∈ R ,

no matter what the higher derivatives of x(t) at tk are. Clearly we can zero all under- and overbraced terms inequations (14) to (19) if A`+1,j has a nontrivial left kernel. This is certainly the case if A has more rows thancolumns, i.e., when `+ 1 > j. The left kernel of A`+1,j is the transposed right kernel of (AT )j,`+1. Each rationalmatrix A`+1,j with `+ 1 > j and 1 ≤ j ≤ 6 has rank j by inspection. Therefore the reduced row echelon form ofeach such (AT )j,`+1 has the block form (Ij Rj,`+1−j)j,`+1.For an arbitrary column seed vector y 6= 0 ∈ R`+1−j , the column vector w = [u; y] ∈ R`+1 in Matlab columnvector notation with u = −Ry ∈ Rj lies in the right kernel of AT since

(Ij Rj,`+1−j)j,`+1 [u; y]`+1,1 = −Ry +Ry = Oj,1.

Thus the linear combination of the equations (14) through (19) with the coefficients of w creates a predictive recur-rence relation for xk+1 in terms of xk, xk−1, ..., xk−` and xk with truncation error order O(τ j+2) by construction.This recurrence relation is look-ahead and determines xk+1 from earlier data. This ends the first, the linear alge-braic part of constructing look-ahead finite difference formulas for possible use in discretized ZNN methods.

A recursion formula’s characteristic polynomial determines its convergence and thus its suitability for discretizedZNN methods. More specifically, the convergence of finite difference formulas and recurrence relations like ourshinges on the lay of the roots of their associated characteristic polynomials. This restriction is well known formultistep formulas and also applies to processes such as discretized ZNN recurrences. Convergence requires thatall roots of the formula’s characteristic polynomial lie inside or on the unit disk in C with no repeated roots allowedon the unit circle, see [1, Sect. 17.6.3] e.g..

The second, nonlinear part of our finite difference formula algorithm tries to find look-ahead and convergent dis-cretization formulas by minimizing the maximal modulus root of ’look-ahead’ characteristic polynomials to below1 + eps for a very small threshold 0 ≈ eps ≥ 0 so that they become numerically and practically convergent whilethe polynomials’ coefficient vectors lie in the left kernel of A`+1,j .The seed vector space for (AT )j,`+1 of the first linear algorithm part is R`+1−j . Any seed y therein spawns alook-ahead finite difference scheme of truncation error order O(τ j+2) when its associated ` + 1 characteristicpolynomial coefficients are used for the linear combination of equations (14) to (19) in ZNN. The set of look-ahead characteristic polynomials is not a linear space since sums of such polynomials may or – most often – maynot represent look-ahead finite difference schemes. Therefore we are only allowed to vary the seed vector andnot the associated look-ahead polynomials themselves when trying to minimize their maximal magnitude roots.Hence we must search indirectly instead in a neighborhood of the starting seed y ∈ R`+1−j for extended look-ahead characteristic polynomials with minimal maximal magnitude roots. After y has been extended successfullyto w ∈ R`+1, w remains a polynomial coefficient vector for the characteristic polynomial of a look-ahead finitedifference formula.Our indirect minimization process uses Matlab’s multidimensional minimizer function fminsearch.m until wehave either found a seed with an associated extended characteristic polynomial that is convergent and for whichits coefficient vector lies in the left kernel of the associated A`+1,j matrix, or there is no convergent such formulafrom the chosen seed. fminsearch.m uses the Nelder-Mead downhill simplex method. Nelder-Mead findslocal minima for non-linear functions without using derivatives. It mimics the method of steepest descend andsearches for local minima via multiple function evaluations. Our seed selection process starts with random entryseeds y ∈ R`+1−j with normally distributed entries in each component. The minimization algorithm runs througha double do loop. An outside loop for a number (100, 500, 2,000, 10,000 or 30,000, or ...) of random starting seedsin R`+1−j and an inner loop for 3 to 6 randomized restarts from the latest updated fminsearch w vector if itdoes not create a polynomial with proper roots, i.e., if we did not find a convergent and look-ahead finite differenceformula in w. In these few inner loop repeats, we project the latest unsuccessful vector w onto a point with newlyrandomized entries nearby and use these as a new seed for another minimization run before starting afresh in theouter loop.In our experiments with trying to find convergent and look-ahead discretization formulas of type j s where


s = ` + 1 − j we never succeeded when 1 ≤ s = ` + 1 − j < j. Success always occurred for s = j andbetter success when s = j + 1 or s = j + 2. It is obvious that for s = ` + 1 − j = 1 and any seed y ∈ R1

there is only one normalized look-ahead discretization formula j 1 and it is never convergent. For convergencewe seemingly need more freedom in our seed space R`+1−j than there is in one dimension or even in less thanj-dimensional space.Remark 1 :

The time-varying discretized ZNN method was designed to solve matrix flow problems accurately in real-time and it is being used exclusively in this way today. Yet discretized matrix ZNN methods are built on adifferential equation, the error DE, and they use special multistep finite difference formulas as the centuriesold numerical initial value ODE solvers do by analytic continuation.What is their difference, i.e., can discretized ZNN be used, understood or adapted from analytic continuationmethods for ODEs or is the reverse possible?Neither one is possible. Time-varying matrix flow ZNN methods are a breed all of their own.Given an initial value ODE problem

y′(t) = f(t, y(t)) with y(to) = yo

we use the integral formula ∫ ti+1

ti

y′(t) dt =

∫ ti+1

ti

f(t, y(t)) dt

and obtain the one-step ahead formula

y(ti+1) = y(ti) +

∫ ti+1

ti

f(t, y(t)) dt

for all i. The aim of all ODE solvers is to compute an accurate approximation for each partial integral off from ti to ti+1 and then approximate the value of y at ti+1 by adding its earlier computed value y(ti) to∫ ti+1

tif(t, y(t))dt. One-step methods use ever more accurate integration formulas to achieve accurate table

of values for y if an antiderivative of y is not known, cannot be found in the antiderivative tables or by usingsymbolic integration software. The multi-step formulas of Adams-Bashforth and Adams-Moulton use severalearlier computed y values and integration formulas in tandem in a two-process method of prediction followedby a correction step. The Fehlberg multi-step corrector formula [1, p.464], [3] uses three previous y valuesand five function f evaluations while the Gear corrector formulas [1, p. 480], [4] use up to six previouslycomputed y values and just one function evaluation to obtain y(ti+1) from data at or before time ti.The onus of accuracy for classic general analytic continuation ODE solvers such as polygonal, Runge-Kutta,Heun, Prince-Dormand and embedding formulas lies within the chosen integration formula’s accuracy for thesolution’s data as their errors propagate and accumulate from each ti to the next and so forth in the computedy(tjj) as jj increases. The Adams-..., Fehlberg and Gear multistep predictor-correction methods differ fromone-step formulas. Their corrector steps are designed for stiff DE systems.The ZNN based methods for time-varying matrix problem differ greatly from any of the above analytic con-tinuation methods that compute function data approximations of antiderivatives numerically. ZNN’s funda-mental difference lies in the stipulated (and actually computed and observed) exponential error decay of thesolution. ZNN solutions are rather impervious to noisy inputs, to occasional sensor failures and even to er-roneous random data inputs (in a limited number of entries of the matrix) because an external disturbancewill become invisible quickly in a solution whose error function decreases exponentially. The speed of decayhinges on a feasibly chosen decay constant η in Step 2 and on the error order of the look-ahead convergentdifference formula that is used. While analytical continuation is best used over small intervals of t values,we have run ZNN methods for seemingly near infinite time intervals [0, tf ] with tf = 5, 10, 20, 100, 200, ...seconds and have seen no deterioration in the error, see Figure 2 with 50 Hz sensor inputs for example. Notefurther that time-varying matrix problems can be started from almost any starting value yo in ZNN and themethod will very quickly lock onto the problem’s proper solution.


But ZNN cannot solve ODEs, nor can it solve static matrix/vector equations; at least we do not know howto. On the other hand, path following IVP ODE based methods have been used to solve formulaic matrixequations, see [12, 27, 31] e.g, but these cannot be adapted easily to sensor clocked data inputs.

Our two part look-ahead and convergent difference equation finding algorithm has computed many convergentlook-ahead finite difference schemes for discretized ZNN of all types j s with 1 ≤ j ≤ 6 and j ≤ s ≤ j + 3with truncation error orders between O(τ3) and O(τ8). Convergent look-ahead finite difference formulas wereunavailable before for error orders above O(τ4). Lower error order ones had sparingly occurred before.Discretized Zeroing Neural Network methods and the quest for high order convergent and look-ahead finite differ-ence formulas bring up many open numerical analysis problems:Are there any look-ahead finite difference schemes with s < j in Aj+s,j and minimally more rows than columns?Why or why not?For relatively low dimensions the rational numbers matrix Aj+s,j can easily be checked for full rank when1 ≤ j ≤ 6. Is that true for all j? Has the rational A matrix ever been encountered anywhere else?Every look-ahead polynomial p that we have constructed from any seed vector y ∈ Rs with s ≥ j by our methodhas had precisely one root on the unit circle within 10−15 numerical accuracy. This even holds for non-convergentfinite difference formula polynomials p with some roots outside the unit disk. Do all Taylor expansion matrixAj+s,j based polynomials have at least one root on the unit circle in C? Can this max modulus root have mag-nitude less than 1? For the stability of multistep ODE methods we must have p(1) =

∑pi = 0, see [1, p.473]

for example. Or are there some convergent look-ahead finite difference formulas with all of their characteristicpolynomial roots inside the open unit disk and none on the periphery?For any low dimensional type j s finite difference scheme there apparently are dozens of convergent and look-ahead finite difference formulas for any fixed truncation error order if s ≥ j > 1. What is the most advantageoussuch formula to use in discretized ZNN methods? What properties of a suitable formula improve or hinder thecomputations for time-varying matrix processes?Intuitively we have preferred those finite difference formulas whose characteristic polynomials have relativelysmall second largest magnitude roots. Is that correct and a good strategy for discretized ZNN methods? See thegraphs at the end of [27].

A list of further observations and open problems for discretized ZNN based time-varying matrix eigen methods isincluded in [25].The error contributions to ZNN’s output from the two sources of rounding errors in solving P\q and the truncationerrors of the finite difference formula were mentioned earlier. One other source of errors relates to the appearanceof derivatives such as A(tk) in q(tk) in formulas (9i) and (10i) above. How should one minimize or equilibratetheir effects for the best overall computed accuracy when using recurrence relations with high truncation errororders? High degree backward recursion formulas for derivatives are generally not very good.

3 Models and Applications of Discretized ZNN Matrix MethodsIn this section we develop specific discretized ZNN algorithms for a number of selected time-varying matrixproblems. Moreover we introduce new matrix techniques to develop matrix models into Matlab code and we pointout references.

Our first example deals with the oldest human matrix problem, namely of solving linear equationsAn,nx = b. Thisgoes back well over 6,000 years to Babylon and Sumer on cuneiform tablets that describe Gaussian eliminationtechniques and solve static linear equations Ax = b for small dimensions n.


Figure 1 : Cuneiform tablet (from Yale) with Babylonian methods for solving a system of two linear equations

(I) Time-varying Linear Equations and discretized ZNN :For simplicity, we consider matrix flows A(t)n,n ∈ Cn,n all of whose matrices are invertible on a time intervalto ≤ t ≤ tf ⊂ R. Our chosen model equation is An,n(t)x(t) = b(t) ∈ Cn for the unknown solution vector x(t).The error function is j1 e(t) = A(t)x(t)− b(t) and the error differential equation is

j2 e(t) = A(t)x(t) +A(t)x(t)− b(t) (∗)= −η A(t)x(t) + η b(t) = −η e(t) .

Solving the inner equation (∗) in j2 first for A(x)x(t) and then for x(t) we obtain the the following two differen-tial equations (DEs,) see also [38, (4.4)]

A(t)x(t) = −A(t)x(t) + b(t)− η A(t)x(t) + η b(t)

and j3 x(t) = A(t)−1 (−A(t)x(t) + b(t)) + η b(t))− η x(t) .

To simplify matters we use the simple 5-IFD formula (11i) of Section 2 again in discretized mode with Ak =A(tk), xk = x(tk) and bk = b(tk) to obtain

j4 xk =8xk+1 + xk − 6xk−1 − 5xk−2 + 2xk−3

18τ∈ Cn .

Equating derivatives at time tk yields

j5 18τ · xk = 8xk+1 + xk − 6xk−1 − 5xk−2 + 2xk−3(∗)= 18τ ·

(A−1k (−Akxk + bk + η bk)− η xk

).

Then the inner equation (∗) of j5 gives us the predictive convergent and look-ahead ZNN formula

j6 xk+1 =9

4τ ·(A−1k (−Akxk + bk + η bk)− η xk

)− 1

8xk +

3

4xk−1 +

5

8xk−2 −

1

4xk−3 ∈ Cn .

Since equation j6 involves the matrix inverse A−1k at each time step tk, we propose two different Matlab codesto solve time-varying linear equations for invertible matrix flows A(t). The code tvLinEquatexinv.m in [32]uses Matlab’s matrix inversion method inv.m explicitly at each time step tk as j6 requires, while our secondcode tvLinEquat.m in [32] uses two separate discretized ZNN formulas. Our second code tvLinEquat.msolves the time-varying linear equation with the help of ZNN that computes the inverse of each A(tk) and thus


it solves equation j6 by using two independent and interwoven discretized ZNN iterations. Both methods runequally fast. The first with its explicit matrix inversion is a little more accurate since Matlab’s inv computes smalldimensioned matrix inverses near perfectly with 10−16 errors while ZNN based time-varying matrix inversionsgive us slightly larger errors, losing 1 or 2 accurate leading digits. This is most noticeable if we use low truncationerror order look-ahead finite difference formulas for ZNN and relatively large sampling gaps τ . There are dozensof references when googling ’ZNN for time-varying linear equations’, see also [34] or [42].

(II) Time-varying Matrix Inverses via ZNN :Assuming again that all matrices of a given time-varying matrix flow A(t)n,n ∈ Cn,n are invertible on a giventime interval to ≤ t ≤ tf ⊂ R, we construct a discretized ZNN method that finds the inverse X(t) of each A(t)predictively from previous data so that A(t)X(t) = In, i.e., A(t) = X(t)−1 is our model here. This gives rise tothe error function j1 E(t) = A(t)−X(t)−1 (= On,n ideally) and the associated error differential equationj2 E(t) = A(t)− X(t)−1 .

Since X(t)X(t)−1 = In is constant for all t, d(X(t)X(t)−1)/dt = On,n. And the product rule gives us thefollowing relation for the derivative of time-varying matrix inverses

On,n =d(X(t)X(t)−1)

dt= X(t)X(t)−1 +X(t)X(t)−1 ,

and thus X(t)−1 = −X(t)−1X(t)X(t)−1 .Plugging this into j2 establishes

E(t) = A(t)− X(t)−1 = A(t) +X(t)−1X(t)X(t)−1(∗)= −η A(t) + η X(t)−1 = −η E(t).

Multiplying the inner equation (∗) above by X(t) from the left on both, the left and right hand sides and thensolving for X(t) yieldsj3 X(t) = −X(t)A(t)X(t)− η X(t)A(t)X(t) + η X(t) = −X(t)((A(t) + η A(t)X(t)− η In) .

If – for simplicity – we choose the same 5-IFD look-ahead and convergent formula as was chosen on line (11i) forStep 4 of the ZNN eigendata method in Section 1, then we obtain the analogous equation to (12i) here with (P\q)replaced by the right hand side of equation j3 . Instead of the eigendata iterates zj in (12i) we use the inversematrix iterates Xj = X(tj) here for j = k − 3, ..., k + 1 and obtain

j5 18τ · Xk = 8Xk+1 +Xk − 6Xk−1 − 5Xk−2 + 2Xk−3(∗)= −18τ ·Xk((A(tk) + η A(tk)Xk − η In) .

Solving (∗) in j5 for Xk+1 supplies the complete ZNN recursion formula that finishes Step 6 of the predictivediscretized ZNN algorithm development for time-varying matrix inverses.

j6 Xk+1 = −9

4τ Xk((A(tk) + η A(tk)Xk − η In)− 1

8Xk +

3

4Xk−1 +

5

8Xk−2 −

1

4Xk−3 ∈ Cn,n

This look-ahead iteration is based on the convergent 5-IFD formula of type j s = 2 3 with truncation error orderO(τ4). The formula j6 requires two matrix multiplications, two matrix additions, one backward approximationof the derivative of A(tk) and a short recursion with Xj at each time step.The error function differential equation j3 is akin to the Getz and Marsden dynamic system (without the dis-cretized ZNN η decay terms) for time-varying matrix inversions, see [5] and [7]. Simulink circuit diagrams forthis model and time-varying matrix inversions are available in [41, p. 97].A Matlab code for the time-varying matrix inversion problem is available in [32] as tvMatrixInverse.m. Adifferent model is used in [40] and several others are described in [41, chapters 9, 12].


Remark 2 : (a) This example reminds us that the numerics for time-varying matrix problems may differ greatlyfrom our static matrix numerical approaches. Time-varying matrix problems are governed by different conceptsand follow different best practices.For static matrices An,n we are always conscious of and we remind our students never to compute the inverse A−1

in order to solve a linear equation Ax = b because this is a very expensive proposition and is rightly shunned.But for time-varying matrix flows A(t)n,n it seems impossible to solve time-varying linear systems A(t)x(t) =b(t) predictively without explicit matrix inversions as has been explained in part (I) above. For time-varying linearequations, ZNN methods allow us to compute time-varying matrix inverses and solve time-varying linear equationsin real time, accurately and predictively. What is shunned for static matrix problems may work well for the time-varying variant and vice versa.(b) For each of the example problems in this section an annotated rudimentary ZNN based Matlab code is storedin [32]. For high truncation error order look-ahead convergent finite difference formulas such as 4 5 these codesachieve 12 to 16 correct leading digits predictively for each entry of the desired solution matrix or vector and theydo so uniformly for all parameter values of t after the initial exponential error reduction has achieved this accuracy.

(III) Pseudoinverses of Time-varying Non-square Matrices with Full Rank and without :Here we first look at rectangular matrix flows A(t)m,n ∈ Cm,n with m 6= n that have uniform full rank(A(t)) =min(m,n) for all to ≤ t ≤ tf ⊂ R.Every matrix Am,n with m = n or m 6= n has two kinds of nullspaces or kernels

N(A)r = {x ∈ Cn | Ax = 0 ∈ Cm} and N(A)` = {x ∈ Cm | xA = 0 ∈ Cn} ,

called A’s right and left nullspace, respectively. If m > n and Am,n has full rank n, then A’s right kernel is{0} ⊂ Cn and the linear system Ax = b ∈ Cm cannot be solved for every b ∈ Cm since the number the columnsof Am,n is less than required for spanning all of Rm . If m < n and Am,n has full rank m, then A’s left kernel is{0} ⊂ Cm and similarly not all equations xA = b ∈ Cn are solvable with x ∈ Cm. Hence we need to abandonthe notion of matrix inversion for rectangular non-square matrices and resort to pseudoinverses instead.There are two kinds of pseudoinverse of Am,n, too, depending on whether m > n or m < n. They are alwaysdenoted byA+ and have size n bym ifA ism by n. Ifm > n andAm,n has full rank n, thenA+ = (A∗A)−1A∗ ∈Cn,m is called the left pseudoinverse because A+A = In. For m < n the right pseudoinverse of Am,n with fullrank m is A+ = A∗(AA∗)−1 ∈ Cn,m with AA+ = Im.In either case A+ solves a minimization problem, i.e.,

minx∈Cn

||Ax− b||2 = ||A+b||2 ≥ 0 for m > n and minx∈Cm

||xA− b||2 = ||bA+||2 ≥ 0 for m < n.

Thus the pseudoinverse of a full rank rectangular matrix Am,n with m 6= n solves the least squares problem forsets of linear equations whose system matrices A have nontrivial left or right kernels, respectively. It is easy toverify that (A+)+ = A in either case, see e.g. [19, section 4.8.5]. Thus the pseudoinverse A+ acts similarly to thematrix inverse A−1 when An,n is invertible and m = n. Hence its name.First we want to find the peudoinverse X(t) of a full rank rectangular matrix flow A(t)m,n with m < n. SinceX(t)+ = A(t) we can try to use the dynamical system of Getz and Marsden [5] again and start withA(t) = X(t)+

as our model.

(a) The right pseudoinverse X(t) = A(t)+ for matrix flows A(t)m,n of full rank m when m < n :

The exponential decay stipulation for our model’s error function j1 E(t) = A(t)−X(t)+ makesj2 E(t) = A(t)− X(t)+(∗)= −η A(t) + η X(t)+ = −η E(t) .

Since A(t)X(t) = Im for all t and A(t) = X(t)+ we have

Om,m = d(Im))/dt = d(A(t)X(t))/dt = d(X(t)+X(t))/dt = X(t)+X(t) +X(t)+X(t) .

Thus X(t)+X(t) = −X(t)+X(t) or X(t)+ = −X(t)+X(t)X(t)+ after multiplying (∗) in j2 through byX(t)+ on the right. Updating equation j2 establishes

A(t) +X(t)+X(t)X(t)+ = −η A(t) + η X(t)+ .


Multiplying both sides on the left and right by X(t) then yields

X(t)A(t)X(t) +X(t)X(t)+X(t)X(t)+X(t) = −η X(t)A(t)X(t) + η X(t)X(t)+X(t) .

Since X(t)+X(t) = In we obtain after reordering that

X(t)X(t)+X(t) = −X(t)A(t)X(t)− η X(t)A(t)X(t) + η X(t)X(t)+X(t) .

But XX(t)+ has size n by n and rank m < n. Therefore it is not invertible. And thus we cannot cancel theencumbering left factors for X(t) above and solve the equation for X(t) as would be needed for Step 3. And avalid ZNN formula cannot be obtained from our first simple model A(t) = X(t)+.This example contains a warning not to give up if one model does not work for a time-varying matrix problem.We try try another model equation for the right pseudoinverseX(t) of a full rank matrix flowA(t)m,n withm < n.Using the definition of A(t)+ = X(t) = A(t)∗(A(t)A(t)∗)−1 we start from the revised model X(t)A(t)A(t)∗ =

A(t)∗. With the error function j1 E = XAA∗ − A∗ we obtain (leaving out all time dependencies of t for betterreadability) j2 E = XAA∗ +XAA∗ +XAA− A∗ (∗)

= −η XAA∗ + η A∗ = −η E .

Separating the term with the unknown derivative X on the left of (∗) in j2 , this becomes

X AA∗ = −X(

(A+ η A)A∗ +AA∗)

+ A∗ + η A∗ .

Here the matrix product A(t)A(t)∗on the left-hand side is of size m by m and has rank m for all t since A(t) does.Thus we can find an explicit expression for X(t), namely

j3 X =(−X

((A+ η A)A∗ +AA∗

)+ A∗ + η A∗

)(AA∗)−1 .

The steps j4 , j5 and j6 now follow as before. The Matlab ZNN based discretized code for right pseudoin-verses is tvRightPseudInv.m in [32]. Our code finds right pseudoinverses of time-varying full rank matricesA(t)m,n predictively with an entry accuracy of 14 to 16 leading digits in every position of A+(t) = X(t) whencompared with the pseudoinverse defining formula. In the code we use the 4 5 look-ahead convergent finite dif-ference formula from [23] with the sampling gap τ = 0.0002.Similar numerical results are obtained for left pseudoinverses A(t)+ for time-varying matrix flows A(t)m,n withm > n.(b) The left pseudoinverse X(t) = A(t)+ for matrix flows A(t)m,n of full rank n when m > n :

The starting model is A+ = Xn,m = (A∗A)−1A∗ and the error function j1 E = (A∗A)X −A∗ then leads toj2 E = A∗AX +A∗AX +A∗AX − A∗ = −η A∗AX + η A∗ = −η E .

Solving j2 for X similarly as before yields

j3 X = (A∗A)−1(−(

(A∗ + η A∗) +A∗A)X + A∗ + η A∗

).

Then follow the steps from subpart (a) and develop a Matlab ZNN code with a truncation error order finite differ-ence formula of your own choice.(c) The pseudoinverses X(t) = A(t)+ for matrix flows A(t)m,n with variable rank(A(t)) ≤ min(m,n) :As before with the unknown pseudo inverse X(t)n,m for a possibly rank deficient matrix flow A(t) ∈ Cm,n, weuse the error function j1 E(t) = A(t)−X(t)+ and the error function DE

j2 E(t) = A(t)− X(t)+(∗)= −η A(t) + η X(t)+ = −η E(t) .


For matrix flows A(t) with rank deficiencies the derivative of X+, however, becomes more complicated withadditional terms, see [6, Eq. 4.12] :

X+ = −X+XX+ +X+X+∗X∗(In −XX+) + (Im −X+X)X∗X+∗

X+ (22)

where previously for full rank matrix flows A(t), only the first term above was needed to express X+. Pluggingthe long expression (22) for X+ into the inner equation (*) of j2 we obtainj3 A+X+XX+ −X+X+∗

X∗(In −XX+)− (Im −X+X)X∗X+∗X+ = −η A(t) + η X(t)+

which needs to be solved for X . Unfortunately X appears once in the second term on the left and twice as X∗ inthe third and fourth term of j3 above. Maybe another start-up error function can give better results, but it seemsthat the general rank pseudoinverse problem cannot be easily solved via the ZNN process, unless we learn to workwith Kronecker matrix products. Kronecker products will be used in subparts (VII), (IX) and (VII start-up) below.The Matlab code tvLeftPseudInv.m for ZNN look-ahead left pseudoinverses of full rank time-varying ma-trix flows is available in [32]. The right pseudoinverse code for full rank matrix flows is similar. Recent work onpseudo-inverses has appeared in [18] and [41, chapters 8,9].

(IV) Least Squares, Pseudoinverses and ZNN :

Linear systems of time-varying equations A(t)x(t) = b(t) can be unsolvable or solvable with unique or multiplesolutions and pseudoinverses can help us.If the matrix flow A(t)m,n admits a left pseudoinverse A(t)+n,m then

A(t)+A(t)x(t) = A(t)+b(t) and x(t) = (A(t)+A(t))−1A(t)b(t) or x(t) = A(t)+b(t) .

Thus A(t)+b(t) solves the linear system at each time t and x(t) = A(t)+b(t) is the solution with minimal Eu-clidean norm ||x(t)||2 since according to (22) all other time-varying solutions have the form

u(t) = A(t)+b(t) + (In −A(t)+A(t))w(t) for any w(t) ∈ Cm .

Here ||A(t)x(t)− b(t)||2 = 0 precisely when b(t) lies in the span of the columns of A(t) and the linear system isuniquely solvable. Otherwise minx(||A(t)x(t)− b(t)||2) > 0.Right pseudoinverses A(t)+ play the same role for linear-systems y(t)A(t) = c(t). In fact

y(t)A(t)A(t)∗ = c(t)A(t)∗ and y(t) = c(t)A(t)∗(A(t)A(t)∗)−1 = c(t)A(t)+ .

Here c(t)A(t)+ solves the left-sided linear system y(t)A(t) = c(t) with minimal Euclidean norm.In this subsection we will only work on time-varying linear equations of the form A(t)m,nx(t) = b(t) ∈ Cm form > n with rank(A(t)) = n for all t. Then the left pseudoinverse of A(t) is A(t)+ = (A(t)∗A(t))−1A∗. Theassociated error function is j1 e(t) = A(t)m,nx(t) − b(t). Stipulated exponential error decay gives us the errorfunction DE j2 e = Ax+Ax− b (x)

= −η Ax+ η b = −η e

where we have again left off the time parameter t for clarity and simplicity. Solving (∗) in j2 for x(tk) gives us

j3 xk = (A∗kAk)−1A∗(−(Ak + η Ak)xk + bk − η bk

).

Here the subscripts . . .k remind us that we are describing the discretized version of our Matlab codes where bkfor example stands for b(tk) and so forth for Ak, xk, ... . The Matlab code for the discretized ZNN look-aheadmethod for time-varying linear equations least squares problems for full rank matrix flows A(t)m,n with m > nis tvPseudInvLinEquat.m in [32]. We advise readers to develop a similar code for full rank matrix flowsA(t)m,n with m < n independently.The survey article [11] describes discretized ZNN methods for nine time-varying different matrix optimization


problems such as least squares and constrained optimizations that we treat in subsection (V) below. More can belearned about current and future ZNN methods for time-varying matrix methods through Google searches.

(V) Linearly Constrained Nonlinear Optimization for Time-varying Matrix Flows :

Optimization problems under linear constraints such as

find max f(x) or min f(x) ∈ R so that Ax = b ∈ Rm for Am,n ∈ Rm,n and x ∈ Rn

with static entries can be solved via Lagrange multipliers by solving the Lagrangian differential equation

∇x,λL(x, λ) = 0 with L(x, λ) = f(x)± λ(Ax− b) . (23)

Here ∇... denotes the gradient function. In the time-varying matrix version of Lagrange, gradient vectors will berow vectors that contain the indicated partial derivatives. The constrained extreme points of f are zeros of theLagrangian differential equation (23) and the values of λ are the Lagrange multipliers that solve the problem.We want to solve the Lagrangian DE for time-varying data A(t), x(t), b(t) and λ(t) in the predictive discretizedZNN fashion. I.e., we want to find x(tk+1) and λ(tk+1) from earlier data for times tj with j = k, k − 1, ... in realtime and accurately. For f : Rn → R, column vectors x(t) ∈ Rn, row vectors λ(t) ∈ Rm, and A(t) ∈ Rm,n,we choose the time-varying Lagrange function with the + sign

L(x(t), λ(t)) = f(x(t)) + λ(t)(A(t)x(t)− b(t)) : Rn+m → R .

We need to evaluate the Lagrangian DE for L via the respective gradient vectors for x and λ. Clearly

∇xL(x(t), λ(t)) = ∇x (f(x(t)) + λ(t)(A(t)x(t)− b(t))) = ∇xf(x(t)) + λ(t)A(t) ∈ Rn and

∇λL(x(t), λ(t)) = (A(t)x(t)− b(t))T ∈ Rm (24)

since the derivative of a linear functions such as λ(t)(A(t)x(t) − b(t)) with respect to x is the row vectorλ(t)A(t) ∈ Rn and with respect to λ it is the column vector A(t)x(t)− b(t) ∈ Rn. To combine the two equationsof (24) we set y(t) = [x(t);λ(t)T ] ∈ Rn+m in Matlab column vector notation and define

j1 h(y(t)) =

((∇xL(x(t), λ(t)))T

(∇λL(x(t), λ(t)))T

)=

((∇xf(x(t)))T +A(t)TλT (t))

A(t)x(t)− b(t)

)=

h1(t)

...

...hn+m(t)

as our error function. To find the derivative y(t) of y(t) we use the multi-variable chain rule which establishes thederivative of h(y(t)) as

h(y(t)) = J(h(y(t))) y(t) + ht(y(t)) .

Here

J(h(y(t))) =

∂h1(t))

∂x1· · · ∂h1(t)

∂λm...

...∂hn+m(t))

∂x1· · · ∂hn+m(t))

∂λm

n+m,n+m

and ht(y(t)) =

∂h1(y(t))

∂t...

∂hn+m(y(t))

∂t

are the Jacobian matrix J of h(y(t)) taken with respect to the location vector x(t) = (x1(t), . . . , xn(t)) and theLagrange factor vector (λ1(t), . . . , λm(t)), and the time derivative of h(y(t)), respectively. For the time-varyingleast squares problem under the standard Lagrangian ’Ansatz’ we stipulate j2 h(y(t)) = −η h(y(t)) which willlead us exponentially fast to the optimal solution y(t) for to ≤ t ≤ tf . Solving for y(t) gives us

j3 y(t) = −J(h(y(t)))−1(η h(y(t)) + ht(y(t))

).


Using the 5-IFD look-ahead finite difference formula once again, this time for yk with discretized data yk = y(tk),we obtain the following derivative free equation for the iterates yj with j ≤ k by equating the two expressions foryk in 5-IFD and in j3 as follows :

j5 18τ · yk = 8yk+1 + yk − 6yk−1 − 5yk−2 + 2yk−3(∗)= −18τ · J(h(yk))−1

(η h(yk)) + ht(yk)

)= 18τ · yk.

Solving (∗) in j5 for yk+1 supplies the complete discretized ZNN recursion formula that finishes Step 6 of thepredictive discretized ZNN algorithm development for time-varying constrained non-linear optimizations via La-grange multipliers:

j6 yk+1 = −9

4τ · J(h(yk))−1

(η h(yk)) + ht(yk)

)− 1

8yk +

3

4yk−1 +

5

8yk−2 −

1

4yk−3 ∈ Cn+m .

The Lagrange based optimization algorithm for multivariate functions and constraints is coded for one specificexample with m = 1 and n = 2 in tvLagrangeOptim2.m, see [32]. For this specific example the optimal so-lution is known. The code can be modified for optimization problems with more than n = 2 variables and for morethan m = 1 constraint functions. It is modular and accepts all look-ahead convergent finite difference formulasfrom [23] that are listed in Polyksrestcoeff3.m in our j s format. It is important to try and understand theinteraction between feasible η and τ values for discretized ZNN methods here in order to be able to use ZNN wellfor other time-varying discretized problems.An introduction to constrained optimization methods is available at https://www.researchgate.net/publication/255602767_Methods_for_Constrained_Optimization ; see also [11]. Several op-timization problems are studied in [11] such as Lagrange optimization for unconstrained time-varying convex non-linear optimizations called U-TVCNO and time-varying linear inequality systems called TVLIS. The latter will betreated in subpart (VI) just below.

Remark 3 : An important concept in this realm is the product τ · η for any one problem and any discretized ZNNmethod which uses a suitable finite difference scheme of type j s. This product of constants, regularly denotedas h = τ · η, seems to be nearly constant for the optimal choice of parameters over a wide range of sampling gapsτ if the difference formula of type j s stays fixed. Yet the optimal value of the near ’constant’ h varies widelyfrom one look-ahead convergent finite difference formula to another. The reason for this behavior is unknownand worthy of further studies; see the optimal η for varying sampling gaps τ tables for time-varying eigenvaluecomputation in [27]. It is interesting to note that analytic continuation ODE methods also deal with optimal boundsfor a product, namely of the stepsize τ and the local Lipschitz constant Li of the solution y, see [1, p. 429].

Thus far in this practical section, we have worked through five models and a variety of time-varying matrix prob-lems. We have developed seven detailed Matlab codes. Our numerical codes implement ZNN look-ahead con-vergent difference formula based discretized processes for time-varying matrix and vector problems in seven stepsas outlined in Section 1. Each of the resulting ZNN computations requires a linear equations solve or an inversematrix times vector product and a simple convergent recursion. Some of the codes are very involved such as forexample (V) which relies on Matlab’s symbolic toolbox and its differentiation functionality. Others were straight-forward. All of our seven algorithms are look-ahead and rely only on earlier data to predict future solutions. Theydo so with high accuracy and run in fractions of a second over time intervals that are 10 to 100 times longer thantheir run time. This adaptability makes discretized ZNN methods highly useful for real-time and on-chip imple-mentations.We continue with further examples and restrict our explanations of discretized ZNN methods to the essentials fromnow on. We also generally refrain from coding further ZNN based programs now, except for one code in subpart(VII start-up) that benefits from a Kronecker product formulation and where we explain how to generate start-updata from completely random first settings via ZNN. We include extended references and encourage our readersto try and implement their own ZNN Matlab codes for their specific time-varying matrix or vector problems alongthe lines of our detailed examples (I) ... (V) above and the subparts (VI) ... (X) that follow.

https://www.researchgate.net/publication/255602767_Methods_for_Constrained_Optimization

https://www.researchgate.net/publication/255602767_Methods_for_Constrained_Optimization


(VI) Time-varying Linear Equations with Linear Equation and Inequality Constraints :

We consider two types of linear equation and linear inequality constraints here :

(A) and (AC)

A(t)m,nx(t)n ≤ b(t)m A(t)m,nx(t)n = b(t)m

C(t)k,nx(t)n ≤ d(t)k .

We assume that the matrices and vectors all have real entries and that the given inequality problem has a uniquesolution for all to ≤ t ≤ tf ⊂ R. Otherwise with subintervals of [to, tf ] in which a given problem is unsolvableor has infinitely many solutions, the problem itself would become subject to potential bifurcations and thus wellbeyond the scope of this introductory survey paper.

In both situations (A) or (AC) it is customary to introduce a nonnegative ’slack variable’ u.2

(t), see [20] e.g., inorder to replace the linear system with inequalities by a system of linear equations. The slack variable u typicallyhas nonnegative entries in the form of squares, i.e., u.

2

(t) = [u21(t); ...;u2`(t)]T ∈ R` with ` = m or k, depend-

ing on the type of time-varying inequality system. By using the special form of u, our models (A) and (AC) become

for (Au) and for (ACu)

A(t)m,nx(t)n + u.2

(t)m = b(t)m

(Am,n Om,kCk,n diag(u)

)m+k,n+k

(xnuk

)n+k

=

(bmdk

)∈ Rm+k.

Here diag(u) denotes the k by k diagonal matrix with the entries ui of u ∈ Rk on its diagonal. The error functionfor (Au) is j1 E(t) = A(t)x(t) + u.

2

(t) − b(t). The product rule of differentiation, applied to each componentfunction u2i (t) of u.

2

(t) establishes the error function DE for (Au) as

j2 E = Ax+Ax+ 2(ui) .∗ (ui)− b(∗)= −η (Ax+ u.

2

− b) = −η E

where the .∗ product uses the Matlab notation for entry-wise vector multiplication. If the unknown entries ofx ∈ Rn and u ∈ Rm are gathered in one extended column vector [x(t);u(t)] ∈ Rn+m we obtain an alternate errorfunction DE for (Au) in block matrix form as

j2 a E = Ax+(A 2 diag(u)

)m,2m

(xu

)2m

− b (∗)= −η (Ax+ u.

2

− b) = −η E ∈ Rm .

Similarly for (ACu), the error function is j1 u Eu(t) =

(A OC diag(u)

)m+k,n+k

(xu

)n+k

−(bd

)∈

Rm+k and its error function DE is j2 u

Eu(t) =

(A O

C 2 diag(u)

)(xu

)−(b

d

)(∗)= −η

((A OC diag(u)

)(xu

)−(bd

))= −η Eu(t).

Solving the error function DEs j2 a and j2 u for the derivative vectors [x(t); u(t)], respectively, via the built-inpseudoinverse function pinv.m of Matlab, see subsection (III) above, we obtain the following expressions for thederivative of the unknown vectors x(t) and ui(t).For model (Au)

j3 a

(xu

)= pinv((Am,n 2 diag(u))m,n+m)n+m,m ·

(b− Ax− η (Ax+ u.

2

− b))m∈ Rn+m ,

and for (ACu)

j3 u

(xu

)= pinv

(A O

C 2 diag(u)

)·((

b

d

)− η

((A OC diag(u)

)(xu

)−(bd

)))∈ Rn+k .


The Matlab function pinv.m in j3 uses the Singular Value Decomposition (SVD). The derivative of the vector[x(t);u(t)] can alternately be expressed in terms of Matlab’s least square function lsqminnorm.m in

j3 als

(xu

)= lsqminnorm

(((Am,n 2 diag(u))m,n+m)n+m,m,

(b− Ax− η (Ax+ u.

2

− b)))m∈ Rn+m ,

or

j3 uls

(xu

)= lsqminnorm

((A O

C 2diag(u)

),

((b

d

)− η

((A OC diag(u)

)(x

(ui)

)−(bd

)))).

Next choose a look-ahead finite difference formula of type j s for the discretized problem and equate its derivative[x(tk); u(tk)] with the above value in j3 a, j3 als or j3 u, j3 uls in order to eliminate the derivatives from now on.Then solve the resulting derivative free equation for the 1-step ahead unknown [x(tk+1);u(tk+1)] at time tk+1.The Matlab coding of a ZNN based discretized algorithm for time-varying linear systems with equation or inequal-ity constraints can now begin after j + s initial values have been set.Recent work on discretized ZNN methods for time-varying matrix inequalities is available in [11, TVLIS], [42]and [34].

(VII) Square Roots of Time-varying Matrices :

Square rootsXn,n ∈ Cn,n exist for all nonsingular static matricesA ∈ Cn,n, generalizing the fact that all complexnumbers have square roots over C. Like square roots of numbers, matrix square roots may be real or complex. Forsingular matrices A the existence of square roots depends on A’s Jordan block structure and its nilpotent Jordanblocks J(0) and some matching dimension conditions thereof, see e.g. [2, p. 466, Thm. 4.5, Cor. 11.3] for details.Here we assume that our time-varying flow matrices A(t) are nonsingular for all to ≤ t ≤ tf ⊂ R. Our modelis A(t) = X(t) ·X(t) for the unknown time-varying square root X(t) of A(t). Then the error function becomesj1 E(t) = A(t)−X(t) ·X(t) and the error DE under exponential decay stipulation is

j2 E = A− XX −XX (∗)= −η (A−XX) = −η E

where we have again omitted the time variable t for simplicity. Rearranging the central equation (∗) in j2 withall unknown X terms on the left-hand side gives usj3 XX +XX = A+ η (A−XX) .

Equation j1 is model (10.4) in [43, ch. 10] except for a minus sign. In j3 we have a similar situation as wasencountered in Section 1 with equation (4) for the time-varying matrix eigenvalue equation, namely the unknownmatrix derivative X appears as both, a left and right factor of matrix products. In Section 1 we switched our modeland solved the time-varying matrix eigenvalue problem one eigenvector and eigenvalue pair at a time. If we usethe Kronecker product for matrices and column vectorized matrix representations in j3 – we could have done thesame with equation (4) of Section 1 for the complete time-varying matrix eigendata problem – then this model canbe solved directly. Here we continue with this model and discretized ZNN by relying on notions from static matrixtheory such as Kronecker products. Classical matrix theory will help us to construct a discretized ZNN algorithmfor finding time-varying matrix square roots.For two real or complex matrices Am,n and Br,s of any sizes, the Kronecker product is defined as the matrix

A⊗B =

a1,1B a1,2B . . . a1,nB

a2.1B. . . a2,nB

.... . .

...am,1B . . . . . . am,nB

m·r,n·s

.


The command kron(A,B) in Matlab creates A ⊗ B for any pair of matrices. Compatibly sized Kroneckerproducts are added entry by entry just as matrices are. For matrix equations a very useful property of the Kroneckerproduct is the rule

(BT ⊗A)X(:) = C(:) where C = AXB . (25)

Here the symbolM(:) denotes the column vector storage mode of Matlab of any matrixM . When we combine theKronecker product with Matlab’s column vector matrix notation X(:) we can rewrite the left hand side of equationj3 XX +XX = A+ η A− η XX as

(XT (t)⊗ In + In ⊗X(t))n2,n2 · X(t)(:)n2,1 ∈ Cn2

,

while its right hand side translates into

A(t)(:)n2,1 + η A(t)(:)n2,1 − η (XT (t)⊗ In)n2,n2 ·X(t)(:)n2,1 ∈ Cn2

.

And miraculously the difficulty of X appearing on both sides as a factor in the left-hand side matrix productsin j3 is gone. We generally cannot tell whether the sum of Kronecker products in front of X(t)(:) above isnonsingular. But if we assume it is, then we can solve j3 for X(t)(:).

X(t)(:) = (XT (t)⊗ In + In ⊗X(t))−1 ·(A(t)(:) + η A(t)(:)− η (XT (t)⊗ In) ·X(t)(:)

). (26)

Otherwise ifXT (t)⊗In+In⊗X(t) is singular, simply replace the matrix inverse by the pseudoinverse pinv(XT (t)⊗In + In ⊗X(t)) above and likewise in the next few lines. A recent paper [37] has dealt with a singular Kroneckerproduct that occurs when trying to compute the Cholesky decomposition of a positive definite matrix via ZNN.With

P (t) = (XT (t)⊗ In + In ⊗X(t)) ∈ Cn2,n2

when assuming non-singularity and

q(t) = A(t)(:) + η A(t)(:)− η (XT (t)⊗ In) ·X(t)(:) ∈ Cn2

we now have X(t)(:)n2,1 = P (t)\q(t). This formulation is reminiscent of formula (10i) in Section 1, except thathere the entities followed by (:) represent column vector matrices instead of square matrices and these vectors nowhave n2 entries instead of n in (10i). This might lead to execution time problems for real-time applications if thesize of the original system is in the hundreds or beyond, while n = 10 or 20 should pose no problems at all. Howto mitigate such size problems, see [14] e.g..To obtain the derivatives X(tk) for each discrete time tk = to+ (k− 1)τ for use in Step 5 of discretized ZNN, weneed to solve the n2 by n2 linear system P (t)\q(t) and obtain the column vectorized matrix X(tk)(:)n2,1. Thenwe reshape X(t)(:)n2,1 into square matrix form via Matlab’s reshape.m function. Equation j5 then equatesthe matrix version X(tk)n,n with the formula for X(tk)n,n from our chosen finite difference expression and thishelps us to predict X(tk+1)n,n. This has been done many times bfore in this paper without enlarged Kroneckerproduct matrices and vectors and this should create no problems to our readers.For further analyses, a convergence proof and numerical tests of this ZNN based time-varying matrix square rootalgorithm, see [43]. Computing time-varying matrix square roots is the subject of [41, Chs. 8, 10].

(VIII) Applications of 1-parameter Matrix Flow Results to Solve a Static Matrix Problem :Concepts and notions of classical matrix theory help us often with time-varying matrix problems. The conceptsand results of the time-varying matrix realm can likewise help with classic, previously unsolvable fixed matrixtheory problems and applications. Here is one example:Numerically the Francis QR eigenvalue algorithm ’diagonalizes’ every square matrix A over C in a backwardstable manner. It does so for diagonalizable matrices as well as for derogatory matrices, regardless of their Jordanstructure or of repeated eigenvalues. QR finds a backward stable ’diagonalizing’ eigenvector matrix similarity


for any A. For matrix problems such as least squares, the SVD, or the field of values problem that are unitarilyinvariant, classic matrix theory does not know of any way to unitarily block diagonalize fixed entry matricesA ∈ Cn,n. If such block decompositions could be found computationally, unitarily invariant matrix problemscould be decomposed into subproblems and thereby speed up the computations for decomposable matrices A.An idea that was inspired by studies of time-varying matrix eigencurves in [28] can be adapted to find unitary blockdecompositions of fixed time-invariant matrices A. [30] deals with general 1-parameter matrix flows A(t) ∈ Cn,n.If X diagonalizes one flow matrix A(t1) via a similarity X−1...X and X−1A(t2)X is block diagonal for somet2 6= t1 with A(t1) 6= A(t2), then every A(t) is simultaneously block diagonalized by X and the flow A(t)decomposes uniformly. [31] then applies this matrix flow result to the previously intractable field of values problemfor decomposing matrices A when using path following methods. Here are the details.For any fixed entry matrix A ∈ Cn,n the hermitean and skew parts

H = (A+A∗)/2 = H∗ and K = (A−A∗)/(2i) = K∗ ∈ Cn,nof A generate the 1-parameter hermitean matrix flow

FA(t) = cos(t)H + sin(t)K = (FA(t))∗ ∈ Cn,nfor all angles 0 ≤ t ≤ 2π. If FA(t1) and FA(t2) are block diagonalized simultaneously by the same unitary matrixX into the same block pattern for someA(t2) 6= A(t1), then every matrix FA(t) of the flow FA is uniformly blockdiagonalized by X and subsequently so is A = H + iK itself.The matrix field of values (FOV) problem [10] is invariant under unitary similarities. The field of values boundarycurve of any matrix A can be determined by finding the extreme real eigenvalues for each hermitean FA(t) with0 ≤ t ≤ 2π and then evaluating certain eigenvector A-inner products to construct the FOV boundary points in thecomplex plane. One way to approximate the FOV boundary curve is to compute full eigenanalyses of FA(tk) fora large set of angles 0 ≤ tk ≤ 2π via Francis QR. Speedier ways use path following methods such as initial valueODE solvers or discretized ZNN methods. But path following eigencurve methods cannot insure that they find theextreme eigenvalues of FA(tk) if the eigencurves of A cross in the interval [0, 2π]. Eigencurve crossings can onlyoccur for decomposable matrices A, see [15]. Finding eigencurve crossings for decomposing matrices A takes upa large part of [12] and still fails to adapt IVP ODE path following methods to all possible types of decompositionsfor static matrices A.The elementary method of [30] solves the field of values problem for decomposable matrices A for the first timewithout having to compute all eigenvalues of each FA(tk) by using Francis QR for example. Our combined ma-trix decomposition and discretized ZNN method is up to 4 times faster than the Francis QR based field of valuesmethod or any other IVP ODE solver method. It depicts the FOV boundary accurately and quickly for all decom-posing matrices A ∈ Cn,n, see [31] for more details and ZNN Matlab codes.

(IX) Time-varying Sylvester and Lyapunov Matrix Equations :

The static Sylvester equation modelAX +XB = C

with A ∈ Cn,n, B ∈ Cm,m, C ∈ Cn,m is solvable for X ∈ Cn,m if A and B have no common eigenvalues.From the error function j1 E(t) = A(t)X(t) + X(t)B(t) − C(t) we construct the exponential decay error DEE(t) = −η E(t) for a positive decay constant η and obtain the equationj2 E(t) = A(t)X(t)+A(t)X(t)+X(t)B(t)+X(t)B(t)−C(t)

(∗)= −η (A(t)X(t)+X(t)B(t)−C(t)) = −η E(t)

and upon reordering the terms in (∗) we havej3 AX + XB = −η (AX +XB − C)− AX −XB + C ,

where we have dropped all references to the time parameter t to simplify reading. Using the properties of Kroneckerproducts and column vector matrix representations as introduced in subsection 2 (VII) above we rewrite the lefthand side of j3 as

(Im ⊗A(t) +BT (t)⊗ In)n·m,n·m · X(t)(:)n·m,1 = M(t)X(t)(:) ∈ Cnm


and the right hand side as

q(t) = −(Im⊗ A+ BT ⊗In)nm,nm ·X(:)nm,1 + C(:)nm,1−η ((Im⊗A+BT ⊗In) ·X(:)−C(:))nm,1 ∈ Cnm .

The Kronecker matrix product is necessary here to express the two sided appearances of X(t) on the left hand sideof j3 . The right hand side of j3 can be expressed more simply in column vector matrix notation without usingKronecker matrices as

q(t) = −((A ·X)(:) + (X · B)(:))nm,1 + C(:)nm,1 − η ((A ·X)(:) + (X ·B)(:)− C(:))nm,1 ∈ Cnm .

Expressions such as (A ·X)(:) above denote the column vector representation of matrix productsA(t) ·X(t). Thuswe obtain the linear system M(t)X(t) = q(t) for X(t) in j3 with M(t) = (Im⊗A(t)+BT (t)⊗ In) ∈ Cnm,nmwhen using either form of q(t). And X(t)(:) ∈ Cnm can be expressed in various forms, depending on the solutionmethod and the state of (non)-singularity of M(t) as X(t)(:) = M(t)\q(t) or

X(t)(:) = inv(M(t)) · q(t) or X(t)(:) = pinv(M(t)) · q(t) or X(t)(:) = lsqminnorm(M(t)), q(t)) ,

with the latter two formulations to be used in case M(t) is singular. Which form of q(t) gives faster or moreaccurate results for X(t)(:) can be tested in Matlab by opening the >> profile viewer and running thediscretized ZNN method for all versions. There are four possible combinations for each of the twoM(t) singularitypossibilities here that users can try and learn how to optimize codes.Once X(t)(:) has been found in column vector form it must be reshaped into an n by m matrix X(t). Next wehave to equate our computed derivative matrix Xk in the discretized version at time tk with a specific look-aheadfinite difference formula expression for Xk in step j5 . The resulting derivative free equation finally is solved forthe future solutionXk+1 of the time-varying Sylvester equation in step j6 of our standard procedure list. Iterationthen concludes the ZNN algorithm for Sylvester problems.

A suitable time-varying Lyapunov equation model is A(t)X(t)A∗(t) −X(t) + Q(t) = On,n for Q(t) = Q∗(t).Its error function is j1 E(t) = A(t)X(t)A∗(t)−X(t)−Q(t)

!= On,n ∈ Cn,n .

Here all matrices are complex and square of size n by n. Now we introduce a shortcut and convert the matrix errorequation j1 immediately to its column vector matrix with Kronecker matrix product formj1 (cvK) E(:) = (A⊗A)X(:)−X(:) +Q(:) ∈ Cn

2

where we have used the formula (AXA∗)(:) = (A ⊗ A)X(:) and dropped all mention of dependencies on tfor simplicity. Working towards the exponentially decaying differential error equation for E(:), we note thatderivatives of time-varying Kronecker products U(t)⊗ V (t) follow the product rule of differentiation

∂(U(t)⊗ V (t))

∂t=∂U(t)

dt⊗ V (t) + U(t)⊗ ∂V (t)

dt

according to [13, p. 486 - 489]. With this shortcut to column vector matrix representation, the derivative of theerror function j1 (cvK) for E(:) is

E(:) = (( ˙A⊗A) + (A⊗ A))X(:) + (A⊗A))X(:)− X(:) + Q(:) ∈ Cn2

And the error function DE E(:) = −η E(:) becomes

j2 E(:) = (A⊗ A− In2)X(:) + ( ˙A⊗A+ A⊗ A)X(:) + Q(:)

(∗)= −η (A⊗A)X(:) + η X(:)− η Q(:) = −η E(:) .


Upon reordering the terms of the central equation (∗) in j2 we have the following linear system for the unknowncolumn vector matrix X(:)j3 (In2 − A⊗A)X(:) = ( ˙A⊗A+ A⊗ A)X(:)− η (In2 − A⊗A)X(:) + η Q(:) + Q(:) ∈ Cn

2

where A is the complex conjugate matrix of A. For M(t) = (In2 − A⊗A) ∈ Cn2,n2 and

q(t)(:) = (˙

A(t)⊗A(t) +A(t)⊗ A(t))X(t)(:)− η (In2 −A(t)⊗A(t))X(t)(:) + η Q(t)(:) + Q(t)(:) ∈ Cn2

we have to solve the system M(t)X(t)(:) = q(t) for X(t)(:) ∈ Cn2

as was explained earlier for the Sylvesterequation. In Step 4 we equate the matrix-reshaped expressions for X in j3 and the chosen look-ahead convergentfinite difference scheme expression for X . Then we solve the resulting derivative free equation for Xk+1 in Steps5 and 6 for discrete data and thereby obtain the discrete time ZNN iteration. These steps, written out in Matlabcommands, give us the computer code for Lyapunov.Introducing Kronecker products and column vector matrix notations early in the construction of discrete ZNN al-gorithms is a significant short-cut for solving matrix equations whose unknownX(t) occurs in time-varying matrixproducts. This is a simple new technique that speeds up discretized ZNN algorithm developments for such matrixequation problems.ZNN methods for time-varying Sylvester equations have recently been studied in [33]. For a new right and left2-factor version of Sylvester see [44]. For recent work on discretized ZNN and Lyapunov, see [21] e.g..

(X) Time-varying Matrices, ZNN Methods and Computer Science :

The recent development of new and predictive ZNN algorithms for time-varying matrix applications has impli-cations for our understanding of computer science and of tiered logical equivalences on several levels in ourmathematical realms.The most stringent realm of math is ’pure mathematics’ where theorems are proved and where, for example, asquare matrix either has a determinant equal to 0 or it has not.In the next, the mathematical computations realm with its floating point arithmetic, zero is generally computedinaccurately as not being 0 and any computed value with magnitude below a threshold such as a small multipleor a fraction of the machine constant eps may be treated rightfully as 0. In the computational realm the aim is toapproximate quantities to high precision, including zero and never worrying about zero exactly being 0.A third, the least stringent realm of mathematics belongs to the engineering world. There one needs to find solu-tions that are good enough to approach the ”true theoretical solution” of a problem as known from the ’pure’ realmasymptotically; needing possibly only 4 to 5 or 6 correct leading digits for a successful algorithm.The concept of differing math-logical equivalences in these three tiers of mathematics is exemplified and inter-preted in Yunong Zhang and his research group’s recent paper [45] that is well worth reading and contemplatingabout.

(VII start-up) How to Implement ZNN Methods On-chip for Sensor Data Inputs and Remote Robots :

Let A(tk) denote the sensor output that arrives at time to ≤ tk = to + (k− 1)τ ≤ tf for a given matrix ’problem’.Assume further that this data arrives with the standard clock rate of 50 Hz at the on-board chip of a robot and thatthere is no speedy access to software such as Matlab as the robot itself may be autonomously running on Mars.At time instance tk the ’problem’ needs to be solved on-chip predictively in real-time well before time tk+1. Wemust predict or compute X(tk+1) on-chip with its limited resources and solve the given problem (at least approx-imately) before tk+1 arrives.

How can we generate start-up data for a discretized ZNN method and the relatively large constant sensor samplinggap τ = 0.02 sec = 1/50 sec that is standardly used in real-world applications, from start-up data that needs to becreated out of thin air. For real-time sensor data flows A(t), we assume that the robot has no information aboutthe ’theoretical solution’ or that there may be no known ’theoretical solution’ at all. Without any usable a-priorisystem information, we have to construct the j + s initial values for the unknown x(tk) or X(tk) and k ≤ j + s


that will be used to iterate with a j s look-ahead convergent finite difference scheme based discrete ZNN method.After many tries at this nearly impossible task, our best choice for the first ’solution’ turned out to be a randomentry matrix or vector for x(to) or X(to) and then iterating through j + s simple low truncation error order ZNNsteps until a higher truncation order predictive ZNN method can take over for times tk, running more accuratelywith truncation error order O(τ j+2) when k � j + s.Here we illustrate this random entries start-up process for the time-varying matrix square root example of solvingA(t)n,n = X(t) ·X(t) ∈ Cn,n from subpart (VII) by using

(a) the Kronecker form representation of X(t)(:) in (26),(b) the matrix column vector notation X(:) ∈ Cn2

for n by n matrices X , and(c) the Kronecker product rule (25) (BT ⊗A)X(:) = C(:) for compatible matrix triple products C = AXB.The Kronecker product representation requires two different notations of matrices here, one as a square array,denoted by the letter m affixed to the matrix name such as in Xm, and another as a column vector matrix, denotedby an added v to the matrix name as in Xv. With these notations, equation (26) now reads as

Xv = (XmT ⊗In+In⊗Xm)−1 ·(Av + η (Av − (In ⊗Xm) ·Xv)

)(26v)

where we have again dropped all mentions of the time parameter t for ease of reading.As convergent look-ahead finite difference formula at start-up we use the simple Euler rule of type j s = 1 2

x(tk) =x(tk+1)− x(tk)

τ(27)

which gives us the expression x(tk+1) = x(tk)− τ x(tk). When applied to the solution matrix X(t) and combinedwith (26v), we obtain the explicit start-up iteration rule

Xv(tk+1) = Xv(tk) + τ · (XmT ⊗ In + In ⊗Xm)−1 ·(Av(tk) + η (Av(tk)− (In ⊗Xm(tk)) ·Xv(tk))

)where (In ⊗ Xm(tk)) · Xv(tk) expresses the matrix square Xm(tk) · Xm(tk) according to the Kronecker tripleproduct rule (25), namely Xm(tk) ·Xm(tk) · I = (I ⊗Xm(tk)) ·Xv(tk). Every iteration step in the discretizedZNN algorithm is coded like this equation exactly as we have shown many times before, with different expressionsof Av and a different look-ahead convergent finite difference formula of type j s accordingly in the main iterationsphase when k > j + s.The Matlab m-file tvMatrSquareRootwEulerStartv.m for finding time-varying matrix square roots ’fromscratch’ is available together with two auxiliary m-files in [32].

To plot the error graph in Figure 2 with our code we compute 18,000 time-varying matrix square roots predictivelyfor 360 sec or 6 minutes, or one every 50th of a second. This process takes around 1.45 seconds of CPU timewhich equates to 0.00008 seconds for each square root computation and shows that our discretized ZNN methodis very feasible to run in real-time for matrix square roots during each of the 0.02 second sampling gap intervals.Note that any ZNN method that is based on a finite difference scheme j s such as 4 5 with j = 4 has a truncationerror order of O(τ j+2) = O(0.02 6) ≈ O(6.4 · 10−11). In this example the model equation A(t) = X(t) ·X(t) issatisfied from a random start after approximately 10 seconds with smaller than 10−12 relative errors. The model’srelative errors decrease to below 10−13 after about 4 minutes according to Figure 2.As τ = 0.02 sec is the standard fixed 50 Hz clocking cycle for sensor based output, it is important to adjust thedecay constant η appropriately for convergence: If a look-ahead ZNN method diverges for your problem and oneη, reduce η; if the error curve decays at first, but has large variations after the early start-up phase and 10 or 20seconds have passed, increase η. The optimal setting of η depends on the data input A(tk) and its variations size,as well as the chosen finite difference formula. The optimal setting for η cannot be predicted. This is one of themany open problems with time-varying matrix problems and both discretized and continuous ZNN methods thatneeds numerical analysis understanding.


Figure 2 : Typical relative error output for tvMatrSquareRootwEulerStartv.m

Eight different models for the time-varying matrix square root problem are detailed in [41, Ch. 10]. Anyone ofthese could have been used here instead.

4 ConclusionsThis paper has introduced a recent new branch of numerical analysis for discretized time-varying, mostly matrixdynamical systems to the West. Time-varying discretized ZNN algorithms are built from standard concepts andwell known relations and facts of algebra, matrix theory, and of static numerical matrix analysis. Yet they differmost deeply from their close cousins, the differential equation initial value solvers. Zhang Neural Networks also donot follow the modern call for backward stable computations that find the exact solution of a nearby problem whosedistance from the given problem depends on the problem’s conditioning. Instead Zhang Neural Networks com-pute highly accurate future solutions based on an exponentially decaying error function that ensures their globalconvergence. In their coded versions, the discretized ZNN algorithms in this paper use just one linear equationssolve and a short recursion of earlier systems data per time step, besides some auxiliary functions. These codes runextremely fast using previous data immediately after time tk and they arrive well before time tk+1 at an accurateprediction of the unknown variable(s) at the next time instance tk+1.Here we have outlined the inner workings of one version of discretized ZNN methods in seven steps. These stepsdo not compare in any way to IVP ODE solvers and are hard to fathom at first. Therefore we have explained manytime-varying problems from matrix theory and optimization from the ground up for ZNN, including working Mat-lab codes for most of them. This was done with ever increasing levels of difficulty and complexity, from simpletime-varying linear equations solving routines to time-varying matrix inversion and time-varying pseudoinverses;from time-varying Lagrange multipliers for function optimization to more complicated matrix problems such as thetime-varying matrix eigenvalue problem, time-varying linear equations with inequality constraints, time-varying


matrix square roots, and time-varying Sylvester and Lyapunov equations. Some of these require Kronecker productmatrix representations and all are handled successfully in the discretized ZNN standard seven steps way.On the way we have encountered models and error functions that do not yield usable derivative information forthe unknown variable(s) and we have seen how to re-define models and error functions accordingly for success.We have pointed out alternate ways to solve the linear equations part of discretized ZNN differently in Matlab andshown how to optimize the speed of some ZNN Matlab codes. We have dealt with simple random entry early on-chip ’solutions’ for unpredictable sensor data as starting values of the discretized ZNN iterations for time-varyingmatrix square roots in light of Kronecker products. But we have not dealt with the final, the engineering applicationof discretized time-varying sensor driven ZNN algorithms or created on-chip circuit designs for use in the controlof chemical plants and in robots, autonomous vehicles and other machinery. That was the task of [41] and it isamply explained in Simulink schematics there. Circuit diagrams often appear in the Chinese engineering literaturethat is quoted here.There are many glaringly open questions with ZNN as, for example, how to choose among dozens and dozens ofotherwise equivalent same j s type look-ahead and convergent finite difference formulas for accuracy or speedand also regarding the ability of specific finite difference formulas to handle widely varying sampling gaps τ well.Neither do we know how to assess or distinguish between high optimal h = τ · η and low optimal h value finitedifference formulas, nor why there are such variations in h for equivalent truncation error order formulas. Theseare open challenges for experts in difference equation.Many other open questions with discretized ZNN are mentioned here and in some of the quoted ZNN papers.

ThanksMy involvement with Zhang Neural Networks began in 2016 when I was sent the book [41] by Yunong Zhang andDongsheng Guo for review, see Zbl. 1339.65002.I was impressed by their work, made contact with Yunong and visited him and his research group at Sun Yat-Sen University in Guangzhou in the summer of 2017. Thanks to the visit and through subsequently exchanginge-mails and advice back and forth, I picked up this totally new-to-me area of predictive time-varying matrix nu-merics. When writing and submitting papers on ZNN to western matrix, numerical, and applied math journals,I soon learned that time-varying numerical matrix methods and predictive solutions thereof had been completelyuntouched in the west. This 20 years old, yet new area has come to us from the outside. It resides and flourishesoutside of our knowledge and understandings base, with around 90+ % of its 400 ± research papers and all of its5+ books originating in China.Two summers ago (2019) I took a quick side trip to visit Nick Trefethen and Yuji Nakatsukasa in Oxford to conferabout the ideas that may lie behind discretized Zhang Neural Networks. We discussed and dissected the basic dis-cretized ZNN algorithm for several hours. This has helped me to come to grips with discretized ZNN algorithmswell enough and put me on the way to try and explain them here.Eventually with the help of referees and editors I recognized discretized ZNN’s seven step set-up structure, startingwith its global entrywise error function for time-varying matrix problems and its stipulated exponential error decaythat makes convergence and noise suppression automatic. Then continuing with an ingenious way to replace theerror differential equation by solving linear equations instead and using look-ahead convergent recursion formulas– that had never been seen or used before – in discretized ZNN’s solution-derivative free setting.These realizations have become the basis for this introductory and partial survey paper. I am glad and thankful forthe help and patience of my family, of my mind; and for the editor’s and referees’ helpful comments, and to Nickand Yuri. and to Peter Benner who early on pointed me to study analytic continuation ODE methods once again.I do hope that time-varying matrix methods and continuous or discretized ZNN methods can become a new part ofour global numerical matrix analysis research and add to our numerical matrix analysis know-how.For this we need to build a new time-varying matrix numerics knowledge base; just as it has been done so manytimes before for our ever evolving matrix computational needs, see [22] e.g..


References[1] GISELA ENGELN–MULLGES AND FRANK UHLIG, Numerical Algorithms with C, with CD-ROM, Springer,

1996, 596 p. (MR 97i:65001) (Zbl 857.65003)

[2] JEAN-CLAUDE EVARD AND FRANK UHLIG, On the matrix equation f(X) = A, Lin. Alg. Appl., 162(1992), p. 447 - 519.

[3] E. FEHLBERG, Numerisch stabile Interpolationsformeln mit gunstiger Fehlerfortpflanzung fur Differential-gleichungen erster und zweiter Ordnung, ZAMM 41 (1961), p. 101 - 110.

[4] C. W. GEAR, Numerical Initial Value Problems in Ordinary Differential Equations, Prentice Hall (1971).

[5] NEIL H. GETZ AND JERROLD E. MARSDEN, Dynamical methods for polar decomposition and inversion ofmatrices, Lin. Alg. Appl., 258 (1997), p. 311 - 343.

[6] GENE H. GOLUB AND VICTOR PEREYRA, The differentiation of pseudo-inverses and nonlinear leastsquares problems whose variables separate, SIAM J. Numer. Anal, 10 (1973), p. 413 - 432.

[7] DONGSHENG GUO AND YUNONG ZHANG, Zhang neural network, Getz-Marsden dynamic system, anddiscrete time algorithms for time-varying matrix inversion with applications to robots’ kinematic control,Neurocomputing, 97 (2012), p. 22 - 32.

[8] DONGSHENG GUO AND YUNONG ZHANG, Zhang neural network for online solution of time-varying linearmatrix inequality aided with an equality conversion, IEEE Trans. Neur. Netw. Learn. Syst., 25 (2014) , p. 370- 382. https://doi.org/10.1109/TNNLS.2013.2275011

[9] L. JIN, Y. ZHANG, Continuous and discrete Zhang dynamics for real-time varying nonlinear optimization,Numerical Algorithms, 73 (2016), p. 11 - 140.

[10] CHARLES R. JOHNSON, Numerical determination of the field of values of a general complex matrix, SIAMJ. Numer. Anal., 15 (1978), p .595 - 602, https://doi.org/10.1137/0715039 .

[11] JIAN LI, YANG SHI AND HEJUN XUAN, Unified model solving nine types of time-varying problems in theframe of zeroing neural network, IEEE Trans. Neur. Netw. Learn. Syst., (2020), in print, 10 p.

[12] SEBASTIEN LOISEL AND PETER MAXWELL, Path-following method to determine the field of values of amatrix at high accuracy, SIAM J. Matrix Anal. Appl., 39 (2018), p. 1726 - 1749.https://doi.org/10.1137/17M1148608 .

[13] JAN R. MAGNUS AND HEINZ NEUDECKER, Matrix differential calculus with applications to simple,Hadamard, and Kronecker products, J. Math. Psych., 29 (1985), p. 474 - 492.

[14] JAMES G. NAGY,http://www.mathcs.emory.edu/˜nagy/courses/fall10/515/KroneckerIntro.pdf .

[15] JOHN VON NEUMANN AND EUGENE PAUL WIGNER, On the behavior of the eigenvalues of adiabatic pro-cesses, Physikalische Zeitschrift, 30 (1929), p. 467 - 470; reprinted in Quantum Chemistry, Classic ScientificPapers, Hinne Hettema (editor), World Scientific (2000), p. 25 - 31.

[16] BINBIN QIU, YUNONG ZHANG AND ZHI YANG, New discrete-time ZNN models for least-squares solutionof dynamic linear equation system with time-varying rank-deficient coefficient, IEEE Transactions on NeuralNetworks and Learning Systems, 29 (2018), p. 5767 - 5776.https://doi.org/10.1109/TNNLS.2018.2805810 .

https://doi.org/10.1109/TNNLS.2013.2275011

https://doi.org/10.1137/0715039

https://doi.org/10.1137/17M1148608

http://www.mathcs.emory.edu/~nagy/courses/fall10/515/KroneckerIntro.pdf

https://doi.org/10.1109/TNNLS.2018.2805810


[17] BINBIN QIU, JINJIN GUO, XIAODONG LI AND YUNONG ZHANG, New discretized ZNN models for solvingfuture system of bounded inequalities and nonlinear equations aided with general ELFS rule, IEEE Transac-tions on Industrial Informatics, to arrear August 2021,https://doi.org/10.1109/TII.2020.3032158

[18] PREDRAG S. STANIMIROVIC, XUE-ZHONG WANG AND HAIFENG MA, Complex ZNN for computing time-varying weighted pseudo-inverses, Applicable Anal. Discr. Math., 13 (2019), p. 131 - 164.

[19] JOSEF STOER AND ROLAND BULIRSCH, Introduction to Numerical Analysis, 2nd edition, Springer, 2002,729 p.

[20] JOSEF STOER AND CHRISTOPH WITZGALL Convexity and Optimization in Finite Dimensions I, Springer,1970, 298 p.

[21] MIN SUN AND JING LIU, A novel noise-tolerant Zhang neural network for time-varying Lyapunov equation,Adv. Diff. Equat., in print (2020), 15 p.

[22] FRANK UHLIG The eight epochs of math as regards past and future matrix computations, Recent Trends inComputational Science and Engineering, S. Celebi (Ed.), InTechOpen (2018),http://dx.doi.org/10.5772/intechopen.73329 , 25 p.[ complete with graphs and references at http://arxiv.org/abs/2008.01900 (2020), 19 p. ]

[23] FRANK UHLIG, The construction of high order convergent look-ahead finite difference formulas for ZhangNeural Networks, J. Diff. Equat. Appl., 25 (2019), p. 930 - 941,https://doi.org/10.1080/10236198.2019.1627343 .

[24] FRANK UHLIG, List of look-ahead convergent finite difference formulas at http://www.auburn.edu/˜uhligfd/m_files/ZNNSurveyExamples under Polyksrestcoeff3.m .

[25] FRANK UHLIG AND YUNONG ZHANG, Time-varying matrix eigenanalyses via Zhang Neural Networks andlook-ahead finite difference equations, Lin. Alg. Appl., 580 (2019), p. 417 - 435,https://doi.org/10.1016/j.laa.2019.06.028 .

[26] FRANK UHLIG, MATLAB codes for time-varying matrix eigenvalue computations via ZNN are available athttp://www.auburn.edu/˜uhligfd/m_files/T-VMatrixEigenv/

[27] FRANK UHLIG, Zhang Neural Networks for fast and accurate computations of the field of values, Lin. andMultilin. Alg., 68 (2020), p. 1894 - 1910, https://doi.org/10.1080/03081087.2019.1648375

[28] FRANK UHLIG, Coalescing eigenvalues and crossing eigencurves of 1-parameter matrix flows, SIAM J Matr.Anal. Appl., 41 (2020), p. 1528 - 1545, https://doi.org/10.1137/19M1286141 .

[29] FRANK UHLIG, The MATLAB codes for plotting and assessing matrix flow block diagonalizations are avail-able at http://www.auburn.edu/˜uhligfd/m_files/MatrixflowDecomp/

[30] FRANK UHLIG, On the unitary block-decomposability of 1-parameter matrix flows and static matrices, Nu-merical Algorithms (2021), https://doi.org/10.1007/s11075-021-01124-7, 21 p; and COR-RECTIONS.

[31] FRANK UHLIG, Constructing the field of values of decomposable and general matrices, submitted, http://arxiv.org/abs/2006.01241, 17 p.

[32] FRANK UHLIG, MATLAB codes for the examples in Section 2 are available athttp://www.auburn.edu/˜uhligfd/m_files/ZNNSurveyExamples/

[33] LIN XIAO, YONGSHENG ZHANG, JIANHUA DAI, JICHUN LI AND WEIBING LI, New noise-tolerant ZNNmodels with predefined-time convergence for time-variant Sylvester equation solving, IEEE Trans. Syst.,Man, Cybern., in print (2020), 12 p.

https://doi.org/10.1109/TII.2020.3032158

http://dx.doi.org/10.5772/intechopen.73329

http://arxiv.org/abs/2008.01900

https://doi.org/10.1080/10236198.2019.1627343

http://www.auburn.edu/~uhligfd/m_files/ZNNSurveyExamples

http://www.auburn.edu/~uhligfd/m_files/ZNNSurveyExamples

https://doi.org/10.1016/j.laa.2019.06.028

http://www.auburn.edu/~uhligfd/m_files/T-VMatrixEigenv/

https://doi.org/10.1080/03081087.2019.1648375

https://doi.org/10.1137/19M1286141

http://www.auburn.edu/~uhligfd/m_files/MatrixflowDecomp/

https://doi.org/10.1007/s11075-021-01124-7



http://www.auburn.edu/~uhligfd/m_files/ZNNSurveyExamples/


[34] FENG XU, ZEXIN LI, ZHUOYUN NIE, HUI SHAO AND DONGSHENG GUO, Zeroing Neural Network forsolving time-varying linear equation and inequality systems, IEEE Trans. Neur. Netw. Learn. Syst., 30 (2019),p. 2346 - 2357.

[35] FENG XU, ZEXIN LI, ZHUOYUN NIE, HUI SHAO AND DONGSHENG GUO, New recurrent neural networkfor online solution of time-dependent underdetermined linear system with bound constraint, IEEE Trans.Industr. Inform., 15 (2019), p. 2167 - 2176.

[36] MIN YANG, YUNONG ZHANG, HAIFENG HU, Relationship between time-instant number andprecision of ZeaD formulas with proofs, Numerical Algorithms, https://doi.org/10.1007/s11075-020-01061-x (2021), 21 p.

[37] XIU ZENG, MIN YANG, JINJIN GUO, YIHONG LING, YUNONG ZHANG, Zhang neurodynamics forCholesky decomposition of matrix stream using pseudo-inverse with transpose of unknown, https://doi.org/10.23919/CCC52363.2021.9549269, IEEEXplore, 40th Chinese Control Conference (CCC)(2021), p.368 - 373.

[38] YUNONG ZHANG AND JUN WANG, Recurrent neural networks for nonlinear output regulation, Automatica,37 (2001), p. 1161 - 1173.

[39] YUNONG ZHANG, YIWEN YANG, NING TAN AND BINGHUANG CAI, Zhang neural network solving fortime-varying full-rank matrix Moore-Penrose inverse, Computing, 92 (2011), p. 97 - 121.https://doi.org/10.1007/s00607-010-0133-9 .

[40] YUNONG ZHANG, ZHAN LI AND KENE LI, Complex-valued Zhang neural network for online complex-valued time-varying matrix inversion, Appl. Math. Comp., 217 (2011), p. 10066 - 10073.

[41] YUNONG ZHANG AND DONGSHENG GUO, Zhang Functions and Various Models, Springer 2015, 236 p.(Zbl 1339.65002)

[42] YUNONG ZHANG, MIN YANG, MIN YANG, HUANCHANG HUANG, MENGLING XIAO AND HAIFENGHU, New discrete solution model for solving future different-level linear inequality and equality with robotmanipulator control, IEEE Trans. Ind. Inform., 15 (2019), p. 1975 - 1984.

[43] YUNONG ZHANG, HUANCHANG HUANG, MIN YANG, YIHONG LING, JIAN LI AND BINBIN QIU, Newzeroing neural dynamics models for diagonalization of symmetric matrix stream, Num. Alg., (2020), in print,18 p.

[44] YUNONG ZHANG, XIAO LIU, YIHONG LING, MIN YANG AND HUANCHANG HUANG, Continuous anddiscrete zeroing dynamics models using JMP function array and design formula for solving time-varyingSylvester-transpose matrix inequality, Numer. Alg., (2020), in print, 24 p.

[45] YUNONG ZHANG, MIN YANG, BINBIN QIU, JIAN LI, MINGJIE ZHU, From mathematical equivalence suchas Ma equivalence to generalized Zhang equivalency including gradient equivalency, Theoretical ComputerScience, 817 (2020), p. 44 - 54.

[ .. /box/local/latex/ZeroingNeuralNetworks2b.tex] October 22, 2021

Also at http://arxiv.org/abs/2008.02724 Aug 6, 2020, Oct 21, 2021

2 image files :

CuneiformYBC4652.pngTVMatrSquaRoot5.png

https://doi.org/10.1007/s11075-020-01061-x

https://doi.org/10.1007/s11075-020-01061-x

https://doi.org/10.23919/CCC52363.2021.9549269

https://doi.org/10.23919/CCC52363.2021.9549269

https://doi.org/10.1007/s00607-010-0133-9

Zeroing Neural Networks, an Introduction to, a Partial ...

Documents

Transcript of Zeroing Neural Networks, an Introduction to, a Partial ...