CompAI - Learning Algorithms for a Neural Network With Laterally Inhibited Receptive Fields - 1998 -...

8/16/2019 CompAI - Learning Algorithms for a Neural Network With Laterally Inhibited Receptive Fields - 1998 - Neural Netw…

1/6

m s

for a Neural

ally I ~ ~ i b ~ ~ e ~

Qiang

Gan* ,

Jun Yao' a nd

K.R.

Subramanian*

School ofElectrical and Elecfronic Engineering

Nanyang Technological University

Nanyang Avenue, Singapore 639

798

E-mail: eqgan@tu. ed us g

Abstract

This pa pe r presents a neural network with its output

layer

as

a

classijier and its hidden layer constrained by

laterally inhibited receptive Pe l& as eature extractor, in

which the idea that wavelet transforms are very suitable

for modeling the primary visual information processing

is

reflected. Two learning algorithms fo r designing the

receptive field s are proposed. The problem associated

with local minima caused by the inherent oscillatory

property in laterally inhibited receptive field s is combated

in the algorithm using discrete wavelets. Good

performance is obtained in the experiment of ECG signal

classification using the neural network.

1 Introduction

Neural networks have been establishedas a general tool

for approximation and classification by fitting

inputloutput

data

effectively into nonlinear models. The

multilayer perceptron, in which a neuron receives inputs

from all the neurons in the adjacent pre-layer, is widely

used for function approximation and signal classification.

On most occasions it performs quite satisfactorily.

However, when the network input

is

characterized by

time-frequency localized features, the generally used

multilayer perceptron with unconstrained global

connections between the adjacent layers does not work

well. In the human visual system there exist example

models for dealing with

this

problem [l]. That is the

conception of receptive field, the shape of which

can

be

adapted with the visual input under certain constraints.In

the human visual system there are various receptive fields

of different shapes. For example, there are Gauss function

shaped receptive fields for local smoothing, and Gabor

Department of Biomedical Engineering

Southeast University

Nanjing 2 I0096

P.R.

China

function shaped receptive fields for combining local

smoothing and sharpening which provides the function of

lateral inhibition. We find that thiskind of receptive fields

can not be formed automatically by learning without any

constraintson the weights

in

a multilayer perceptron.

Wavelet transform is a good model for the receptive

fields in the human visual system [2]. Because a wavelet

function satisfies the admissibility condition

[3],

it mustbe

oscillatory across its zero points. Hence, wavelet hct ions

provide

natural

models for laterally inhibited receptive

fields, which are good at extracting time-frequency

localized features. Actually, Gabor function has been

widely used in theoretical studies of the primary visual

information processing such as ateral inhibition and it can

be regarded as a mother wavelet of good time-frequency

localization properties. Through dilation and translation a

wavelet filter bank can

be

formed as a group of receptive

fields which approximately perform wavelet transforms.

Hence, in the design of receptive fields we can benefit from

the advanced theory of wavelet transforms.

There have been several pieces of work done on

combining neural networkswith wavelet transforms which

perform as the receptive fields of hidden neurons. Szu [4]

developed neural network adaptive wavelets for signal

representation and classification, and tentatively applied

them in phoneme recognition and image compression.

Gan [ 5 ] proposed a wavelet neural network architecture

and applied it to ECG signal classification. Dickhaus [6]

and Akay

[7]

ave a lso studiedbiomedical signal detection

and classification using different wavelet network

structures. The key issue in the design of this kind of

neural networks

is

how to obtain optimal sets of dilation

and translation parameters (or wavelet parameters). In all

the networks mentioned above, continuous wavelet

parameters are used and trained by the gradient-descent

learning algorithm, or preset and fixed wavelets are

0 7803 4859

1/98 O.oOO1998 E E E

1156


2/6

applied. Because of the inherent oscillatory property of the

wavelet function, learning wavelet parameters is easy to

sink into local minima and it is difficult to get the optimal

result. In

this

paper, we use Gabor function o constrain the

receptive fields of the hidden neurons so that the lateral

inhibition is introduced into the network. Furthermore,

discrete wavelets are used and a method for calculating

wavelet parameters is proposed to combat the problem of

unconvergence in the learning process.

The remainder of

this

paper is organized as follows.

The neural network formulation is put forward in

section 2. Two learning algorithms are proposed in section

3.

Simulation studies on ECG signal classification are

carried out in section 4, followed by discussions and

conclusions in section 5 .

2.

Neural network formulation

The neural network under consideration, with laterally

inhibited receptive fields, can be described as follows:

i =

1,2

......,NOL.

s

t )= 1 - ' ) 1

+

e-.'

)

where, y f L and X, denote the output and the input of the

network respectively, represent the weights between

output and hidden layers,

h ( ( j - b 1 ) / u , )

produce the

weights between hidden and input layers, and NOL, NHL,

NIL

are respectively the number of nodes in output,

hidden, and input layers.

0

5

-0.5

5

Figure I abor function and lateral inhibition

Note that the weights connected to the hidden layer are

generated by dilation and translation of a Gabor function

h(4 and can be adjusted by dilation and translation

parameters a, and

b,

.If we regard the Gabor function as

a

mother wavelet, the hidden layer is actually composed of

a

wavelet filter bank which plays the role of feature

extraction. In connection with the human visual system,

the weights connected to the hidden layer

perform

just like

receptive fields with lateral inhibition because the Gabor

function shapes like a Mexican hat as shown in Fig.1.

Compared to a general feedforward neural network, the

neural network described by (1)-(3) has a similar structure,

but the weights connected to the hidden layer

are

constrained and adapted indirectly by learning dilation

and translation parameters.

As we know, Gabor wavelets are nonorthogonal. Gabor

function is selected here as the mother wavelet based on

the following reasons. First, although there are many

orthogonal wavelet bases, the isotropic (or symmetric),

compactly supported and orthogonal wavelets do not exist.

Second, according to the uncertainty principle about the

time-frequency resolution, a Gaussian or modulated

Gaussian provides the optimal tradeoff between time

localization and frequency resolution. However, Gauss

function does not satisfy the admissibility condition and

can not provide lateral inhibition. Gabor function is a

modulated Gaussian, which not only has optimal joint

time-frequency resolution [ti],but also introduces lateral

inhibition. Third, Gabor function

has

been proved

to

be a

very good model for the primary visual information

processing. Furthermore, our purpose of using the

wavelets is to extract useful features, but not to reconstruct

signals.

It is well known that wavelet transforms are good at

representing signals with time-frequency localized

features. The neural network proposed here would be

suitable for the classification of signals or patterns with

time-frequency localized characteristics.

3. Learning algorithms

Learning algorithms are developed in order to obtain

optimal weights for classification and wavelets for

extracting features from a particular type of signals. In the

case when continuous parameters are used, a learning

algorithm can be derivedout based on the gradient-descent

method. This kind of learning algorithm may not converge

when there exist oscillatory functions in the neural

network formulation. We try to combat

this

problem by

using discrete wavelets and learning them by a

constructive algorithm.

3.1 For

receptive fields with continuous

parameters

1157


3/6

If

the wavelet parameters

a,

and

b,

take continuous

values, a back-propagation base

algorithm

can

be

used to

train the neural network defined by (1)-(3), although

trivial modification on the rules for updating weights and

parameters

should be

made. We can derive the updating

rules

as

follows:

In the above equations,

d,

represents the desired output,

y p

is the output of the Ith node in the hidden layer, andE

denotes the error used in the back-propagation.

To

keep the receptive fields laterally

inhibited, the

mother wavelet function

has

to be oscillatory across its

zero points, as shown in Fig. 1. This will make the error

function

E

highly

nonconvex. Therefore, local minima are

expected in the learning process, or the learning process

may not converge. This is the major shortage to be

overcome in training thiskind ofneural networks. We will

resolve this problem by using discrete wavelets in the

following subsection.

3.2

For receptive fields with discrete

parameters

Firstly, the wavelet parameters are constrained to take

discrete values by the following equations:

a, =ao-

6, = n,b,a, =nlboal

(14)

In thisway, the receptive fields are reformed as follows:

where m,

,n, E Z .

Instead of training

a,

and

b,

directly

by error back-propagation, they can be calculated

according to (14)

if

we can determine the values of

a,,b,,m,, andn,. The formula for determining these

parameters are derived in the following.

h,, t) will define sets of windows in a wavelet filter

bank. For brevity, we just consider a particular set of

windows defined by wavelet parameters a and

b,

corresponding to a,,b,,m,

andn.

In the discrete domain,

the windows indexed bym and n and defined by h,,

t)

in

the time-frequency plane can be written as follows:

[nb,af

+ af t* f A h , b,a, + a f t *+ a f A h )

a A, a'

A,

x[--- -+--)

a f a f a r

a,

where, f

and

A,,

are the centre and the radius of the

mother wavelet

h(t),

respectively, while and

A,

are

those of the Fourier transform

i ( w ) ,

respectively.

Actually, t * , Ah,

w , a d

Ah can be determined by

w, nd T which are parameters in the Gabor function.

Selecting suitable values for parameters a, and bo is

important. From the point of view of function

reconstruction, we hope that suitable selection of a, and

bo

can make

h , ( t )

constitute a tight frame [3]. On the

other hand, from the point of view of filter banks, we hope

that the windows defined by

h,, t)

can properly cover the

interesting

areas

in the time-frequency plane. Because we

are interested in extracting useful time-frequency ocalized

features, we will select the values of a, and

bo

from the

point

of view of

filter banks.

If

a,

and

bo

are

not

properly

selected, the windows will be too sparse to cover the whole

1158


4/6

time-frequency plane and the information in signals will

be lost; or the windows will be overlapped and the

information extracted is redundant. From

16)

we note that

the windows in the frequency domain are not influenced by

the translation parameter. Let us consider two adjacent

windows corresponding to m and m+l. In order that the

windows can cover the whole frequency domain, while

having no window overlapped, the following equality

should be satisfied:

In the time domain, the situation

is

more complicated,

because the windows are affected by both dilation and

translation parameters and the number of adjacent

windows are more than two. However, we only pay

attention to the selection of

bo

in the time domain. We

consider two adjacent windows with index m unchanged.

In this case, the windows in time domain should satisfy:

( n+ l)boar+a r t *

-

arA,

= nboa, +

ar t +

a, Ah

,

Le.,

bo

=

2 A h

18)

Although 18) does not give an ideal selection of

b o ,

it

provides a simple and satisfactory solution.

Now let

us

consider how to determine m and

n. As

we

know, the centre of the window defined by

h,(t)

in the

frequency domain is

w / (~zu, ),

nd that in the time

domain is

nb,a, +a, t'. If

the maximum and the

minimum frequencies of the input signals are f,, and

f,, (f,, f 0)

, espectively, and the maximum length of

input signals is

T,,

, hen the following inequalitiesshould

be satisfied

w *

f m i n 5

f

max

0 5 nb,a,

+ar t

5 T,,

20)

Hence, the ranges of

m

and

n

should be constrained as

follows:

>n>---

t * if bo

c0)

r --

8

bo a3 bo

23)

Because the values of

m

and n are limited by 2 1)- 23),

we can even take into account all the integer values of m

and

n

in the whole ranges given above and calculate the

output values of the corresponding wavelet filters. Those

m

and

n

which result in output values larger

t h n

the

given thresholds will be selected. Note that the number of

wavelet filters, or the number of hidden nodes, are

automatically determined by learning.

Given a set of training signals, we can get satisfactory

wavelet parameters using the above algorithm. It

is

able to

avoid the problem of unconvergence which exists in the

continuous wavelet parameter adjustment as described in

4)-

13). It should be noted that the weights connected to

the output layer are still trained using the back-

propagation algorithm.

4. Simulation studies

To

test the classification performance of the neural

network with laterally inhibited receptive fields and the

corresponding learning algorithms, we apply the network

to

ECG signal classification. The hidden layer consists of

three subnetworks, each receiving one period of the ECG

signal respectively. All the outputs of the subnetworks are

connected to an output layer where the final classification

is

accomplished. The network size is set as follows:

NOL=4,

NIL=460x3, and the size of the hidden layer is

determined by the learning algorithm. Three subnetworks

are needed because generally at least three periods of

ECG

signals are required for diagnosing some cardiac disease

[ 5 ] . Because the page length of the paper is limited, the

details of the network architecture are not fully discussed

here. The input-output function of the network

is

given as

follows:

= 42,. ,NOL

where,

k

is the subnetwork index. In the case of applying

continuous wavelets,

a,

and b,

are directly trained

according to

4)- 13). If

discrete wavelets are used,

a, and b, are calculated according to 14), with

ao,bo,m, undn, determined by 17)- 18)and

21)- 23).

1159


5/6

The ECG signals used in our simulation are from the

MIT/BIH arrhythrma

database.

We have selectedNormal,

Bundle Branch Block, Infarct, and Premature Ventricular

Contraction waves, each consistingof

200x3

periods. They

are

divided into a training set and a testing set. Some

typical examples of ECG signals

are

shown in Fig. 2. We

note that the ECG signals are mostly flat and their high

frequency componentsare localized to short time intervals.

For

this

kind of signals, time-frequency localized

representations such aswavelet transforms are very useful.

BP I 85% 88.75% I 99% I 98% ~

Normal

Net type

I

Normal

Bundle Branch Block

B.B.B I Infarct

I

P.V.C

Infarct

NN

LI1 92.5% 95% 198.75%

NN L12 I 100% 96.6% I 98.25%

Premature Ventricular Contraction

90.5%

98.25%

Figure2. Typical ECG signals

Before entering the network, the ECG waves

are

preprocessed. The preprocessing includes the detection of

peaks

of

R-waves

and

the normalization which makes the

amplitudes of ECG signals minus their mean values

change from -1 to l. Three periods of ECG waves are

directly inputted into the network at the same time. The

peaks

of the R-waves are located at the centres of the three

subnetworks respectively.

To

further improve the

classification performance, a centre for each class of ECG

waves can be obtained by a clustering algorithm in the

training phase. The differences between the ECG wave

and the centres can be used as inputs to the network.

With discrete wavelets, in order to make the Gabor

function

satisfy

the admissibility condition of wavelet

transforms,

its parameters are set

as

follows: o2

8,

and

W , 5 3

According to (17) and (18), uo=1.14 and

bo =

3.28. We select integer values of

mkl

and nk, from

the ranges given by (2 1)-(23), on ly those corresponding to

the wavelet filters that obtain output values larger th nthe

given thresholds are maintained to form the receptive

fields, which extract useful time-frequency localized

features of the input signal for classification. The number

of wavelet filters selected in each subnetwork by the

learning procedure is

as

follows:

NHL

15, NHL,

=

16, M L ,

= 15.

After learning, the classification performance of the

network trained with discrete wavelets

is

tested. The

simulation result is given in the row marked

as

NN-LI2 in

Table

1.

In order to make a comparison, the network

trained with continuous wavelets is also investigated

in

the

simulation studies. The corresponding classification

performance is given in the row marked as NN-LI1 in

Table 1. We note that it often happens for the network with

continuous wavelets to sink into local minima or not to

converge. The performance of a standard BP network with

a similar structure is also given in Table 1. It is shown that

the neural networks with laterally inhibited receptive fields

achieve better recognition rate than the BP network. We

note that the neural network proposed in this paper and the

BP network have a similar structure and operate in the

same way in the testing phase, although their weights are

trained by different algorithms. Therefore, the comparison

here is reasonable. Although the ECG signals in the testing

set are corrupted by noise and fake peaks, and have never

been seen by the network before testing, the recognition

rate

is

considerably high.

5. Discussions and conclusions

In the simulation studies, Gauss function shaped

receptive fields have also been tested. It is illustrated that

Gabor function shaped receptive fields perform

better

in

the ECG signal classification. The major difference

between

Gauss

and Gabor functions lies in that Gabor

1160


6/6

function is oscillatory across its zero points and thus

provides lateral inhibition. The smoothing effect of Gabor

function

as

a

receptive field plays a role

of

removing noise,

while

its

sharpening ability resulting from the lateral

inhibition can enhance the localized features in signals.

Obviously, the lateral inhibition provided by Gabor

function shaped receptive fields improves the feature

extraction ability of the neural network. We note that

lateral inhibition is important

,

but it can not be learnt

using BP-type algorithms without any constraints

on

the

weights in advance.

The oscillatory behavior of the Gabor function makes it

satisfy

the admissibility condition of wavelet functions.

The Gabor function shaped receptive fields form a wavelet

filter

bank

and the hidden layer of the neural network plays

a rule similar to a group of wavelet transforms. What is

more, by learning the wavelet parameters it is possible to

achieve the optimal or sub-optimal result.

From the point of view of the generalization ability of

neural networks, we should be able to figure out

theoretically how

the

lateral inhibition influences the

performance surface of the neural network. This would be

one of our future research directions.

In

this

paper, Gabor function shaped receptive fields are

successfully introduced into a neural network for signal

classification. By using discrete wavelets a learning

algorithm

is

derived such that the problem of

unconvergence caused by the oscillatory property of the

receptive fields can be eliminated. Simulation studiesshow

that introducing lateral inhibition into the neural network

is useful to improve its classification performance.

[51

171

R.B.Pinter and B.Nabet, ed., Nonlinear Vision:

Determination of Neural Receptive Fields, Function,

and Networks, CRC Press, 1992.

L.Gaudart, J.Crebassa, and J.P.Petrakian, “Wavelet

transform in human

visual

channels”, Applied

Optics,vo1.32, no.22, 1993,pp.4119-4127.

LDaubechies, Ten Lectures

on

Wavelets, Society for

Industrial and Applied Mathematics, Philadelphia,

1992.

H.Szu, B.Telfer, and S.Kadambe, “Neural network

adaptive wavelets for signal representation and

classification”, Optical Engineering, vo1.3 1, 110.9,

Q.Gan, J.Yao, H.-C.Peng, and J.-Z.Zhou, “Wavelet

neural network for ECG signal classification”, Proc.

of Int. Conf. on Biomedical Engineering, Hong

Kong, 1996.

H.Dickhaus and H.Heinrich, Classifying biosignals

with wavelet networks”, IEEE Engineering in

Medicineand Biology Magazine, vo1.15, no.5, 1996,

Y.M.Akay, M.Akay, W.Welkowitz, and J.Kostis,

“Noninvasive detection of coronary artery disease

using wavelet-based fuzzy neural networks”,

IEEE

Engineering in Medicine and Biology Magazine,

vo1.13, 110.5, 1994, pp.761-764.

J.Ben-Arie and K.R.Rao, “Nonorthogonal signal

representation by Gaussians and Gabor functions”,

E E E Trans. on Circuits and Systems, 11, vo1.42,

no.6, 1995, pp.402-413.

1992, pp.1907-1916.

pp. 103-111.

References

1161

CompAI - Learning Algorithms for a Neural Network With Laterally Inhibited Receptive Fields - 1998 -...

Documents

Transcript of CompAI - Learning Algorithms for a Neural Network With Laterally Inhibited Receptive Fields - 1998 -...