Ms. Shraddha Kadam, Asst. Professor

11/8/2019Compiled by: Ms. Shraddha Kadam-Shah (University Department of Information Technology)

Ms. Shraddha Kadam,

Asst. Professor

[email protected]

[email protected]

Chapter 4: Associative Memory Network REFERENCE: PRINCIPLES OF SOFT COMPUTING (THIRD EDITION)

BY S.N. SIVANANDAM & S.N.DEEPA

Introduction


Introduction

An associative memory network can store a set of patterns as memories.

When the associative memory is being presented with a key pattern, it responds by producing one of the scored patterns, which closely resembles or relates to the key pattern.

Thus, the recall is through association of the key pattern, with the help of information memorized.

Type of memories:

Content addressable memory

Address-addressable memory


Introduction

Associative memory makes a parallel search within a stored data files.

The concept behind this search is to output any one or all stored items which match the given search argument and to retrieve the stored data either completely or partially.

Two types of memories can be differentiated: Autoassociative & heteroassociative memory

Both this nets are single layer nets in which the weights are determined in a manner that the net stores a set of pattern associations.

Each of this association is an input-output vector pair s:t

If each of the output vector is same as the input vectors which it is associated, then the net is said be autoassociative memory net

If the output vector is different from the input vectors then the net is said to be heteroassociative memory net


Introduction


The architecture of the associative net may be either feed-forward or iterative (recurrent)

Training Algorithms for Pattern Associations


Hebb Rule

Step 0: Set all initial weight to zero i.e. wij = 0 (i = 1 to n, j= 1to m)

Step 1: For each training target input output vector pairs s:t, perform Steps 2-4.

Step 2: Activate the input layer units to current training input

xi=si (for i= 1 to n)

Step 3: Activate the output layer units to current target output,

yj=tj (for j=1 to m)

Step 4: Start the weight adjustment

wij(new) =wij(old) + xiyj (i = 1 to n, j= 1to m)

This algorithm is used for the calculation-of the weights of the associative nets. Also, it can be used with patterns that are being represented as either binary or bipolar vectors


Outer Product Rule (Alternative Method)

Input s = (s1….sj…sn)

Output t =(t1….tj…..tm)

The outer product of the two vectors is the product of the matrices s = sT and T = t, i.e., between [n X 1] matrix and [1 x m] matrix.

For storing a set of associations, s(p):t(p), p = 1 toP, wherein, ,(p) = (s1(p), ... , sj(p), ... , sn(p)) &t(p) = (t1(p), · · ·tj(p), · · · tm(p)) & weight matrix W={wij} can be given


Autoassociative Memory Network


Autoassociative Memory Network

In the case of an autoassociative neural net, the training input and the target output vectors are the same.

The-determination of weights of the association net is called storing vectors.

This type of memory net needs suppression of the output noise at the memory output.

The vectors that have been stored can be retrieved from distorted (noisy) input if the input is sufficiently similar to it. The net's performance is based on its ability to reproduce a stored pattern from a noisy input.

lt should be noted, that in the case of autoassociative net, the weights on the diagonal can be set to zero. This can be called as auto associative net with no self-connection.

The main reason behind setting the weights to zero is that it improves the net's ability to generalize or increase biological plausibility of the net. This may be more suited for iterative nets and when delta rule is being used.


Architecture

The input layer consists of n input units and the output layer also consists of n output units.

The input and output layers a re c o n n e c t e d t h r o u g h w e i g h t e d interconnections.

The input and output vectors are perfectly correlated With each other component by component.


Training Algorithm

Step 0: Set all initial weight to zero i.e. wij = 0 (i = 1 to n, j= 1to m)

Step 1: For each training target input output vector pairs s:t, perform Steps 2-4.

Step 2: Activate the input layer units to current training input

xi=si (for i= 1 to n)

Step 3: Activate the output layer units to current target output,

yj=tj (for j=1 to m)

Step 4: Start the weight adjustment

wij(new) =wij(old) + xiyj (i = 1 to n, j= 1to m)

The weight can also be determined by the formula


Testing Algorithm


HeteroassociativeMemory Network


Heteroassociative Memory Network


Architecture

The input layer consists of n number of input uni t s and the output layer consists of m number of output units.

There exist weighted interconnections between the input and output layers.

The input and output layer units are not correlated with each other.


Testing Algorithm


Bidirectional Associative Memory(BAM)


Bidirectional Associative Memory (BAM)

The BAM network performs forward & backward associative searches for stored stimulus responses.

The BAM is a recurrent heteroassociative pattern-matching network that encodes binary or bipolar patterns using Hebbian learning rule.

It associates patterns, say from set A to patterns from set B & vice versa is performed.

BAM neural network can respond to the input from either layer (input layer or output layer)

There exist two types of BAM, called discrete & continuous BAM


Architecture

The network dynamics involve two layers of interaction.

The BAM network iterates by sending the signals back and forth between the two l aye r s un t i l a l l th e ne u ron s re ach equilibrium.

The weights associated with the network are bidirectional.

Thus, BAM can respond to the inputs in either layer.

The layers can be connected in both direction (bidirectional) with the result the weight matrix sent from the X layer to the Y layer is W and the weight matrix for signals sent from the Y layer to the X layer is WT.


Discrete BAM

When the memory neurons are being activated by putting initial vector at the input layer, the network evolves a two-pattern stable state with each pattern at the output of one layer.

The two bivalent forms of BAM are found to be related with each other i.e. binary & bipolar.

The weights in both cases are found as the sum of the outer products of the bipolar form of the given training vector pairs.

A nonzero threshold is been assigned. The activation function is a step function.

The bipolar vectors improve the performance of the net to large extent when compared to binary.


Discrete BAM-Determination of Weights


Discrete BAM- Activation functions


Discrete BAM- Testing Algorithms


Continuous BAM

A continuous BAM transforms the input smoothly and continuously in the range 0-1 using logistics sigmoid functions as the activation functions for all units.

The logistics sigmoidal function may be either binary or bipolar sigmoidal functions

When a bipolar sigmoidal function with a high gain is chosen, then the continuous BAM converge to a state of vectors which will approach vertices of the cube.


Continuous BAM- Activation functions


Analysis of Hamming Distance

The hamming distance is defined as the number of mismatched components of two given bipolar or binary vectors.

It can also be defined as the number of different bits in two binary or bipolar vectors X and X'. It is denoted as H[X,X'].

The average hamming distance between the vectors is (1/n)H[X,X'], where "n" is the number of components in each vector.

Consider the vectors, X: [1 0 1 0 1 1 0] and X': [1 1 1 1 0 0 1]. The hamming distance between these two given vectors is equal to 5. The average hamming distance between the corresponding vectors is 5/7.


Energy Function of BAM


Storage Capacity of BAM


Thank you!!!

Ms.Shraddha Kadam,Asst. Professor

[email protected]@gmail.com


Ms. Shraddha Kadam, Asst. Professor

Documents

Transcript of Ms. Shraddha Kadam, Asst. Professor