Agenda Hopfield Neural Network Gaussian Machine Quadratic Assignment Problems How to Solve Problem Computation Results Conclusion

Artificial Neural Network (ANN) An mathematic model Inspired by the biological nervous systems Acquires knowledge through learning ANN’s knowledge is stored within inter-neuron

connection strengths (Synaptic weights)

Use continuous activation function Fully connected recurrent network Notion of energy Each state has an energy Computes recursively until a stable state

reached Converges to stable states

Hopfield Network Model:

Hopfield Neural Network

1

( ) ( ) ( )N

ii i j i i

j

du t u t T x t Idt =

= - + +å

0

( )1( ) ( ( )) 1 tanh2

ii i

u tx t f u ta

æ öæ ö= = +ç ÷ç ÷ç ÷è øè ø

Dynamic

Equation:

Output:

Hopfield Neural Network

Activation Function

a0 parameter determines behavior of the gain function Higher ~ gentle Lower ~ steep

0

( )1( ) ( ( )) 1 tanh2

ii i

u tx t f u ta

æ öæ ö= = +ç ÷ç ÷ç ÷è øè ø

ui

xi =f(ui)

0

1

Hopfield Network: Energy Function Quantification of current energy of the network Energy surface determines stable states Stable states are local minima

Each update converges to stable state Symmetric connections

2

1

0N

i

i

dxdEdt dt=

æ ö= - £ç ÷è ø

å

1 1 1

1( )2

N N N

i j i j i ii j i

E x T x x I x= = =

= - -åå å

, 0i j ji iiT T T= =

Energy

function:

Lyapunov

Condition:

Gaussian Machine:An Improvement of Hopfield NNGaussian Machine Objective: Allow the system to escape from local minima Added Gaussian noises that its power vary in time Vary the activation function gain in time

,1

( ) ( ) ( ) ( )N

ii i j j i i

j

du t u t T x t I tdt =

= - + + +å

1( ) ( ( )) 1 tanh2 ( )i ix t f u tæ öæ ö

= = +ç ÷ç ÷ç ÷è øè ø

Dynamic

Equation:

Output:

Gaussian Machine:An Improvement of Hopfield NNGaussian Machine Added Gaussian noises that its power vary in time

( ) (0, ( ))i it N th s=

Dynamic

Equation:Temperature

0 0.5 1 1.5 2 2.5 3-1.5

-1

-0.5

0

0.5

1

1.5

t

( )i th

,1

( ) ( ) ( ) ( )N

ii i j j i i

j

du t u t T x t I tdt =

= - + + +å

Gaussian Machine:An Improvement of Hopfield NNGaussian Machine Vary the activation function gain in time

Output:

ui

xi =f(ui)1

1( ) ( ( )) 1 tanh2 ( )i ix t f u tæ öæ ö

= = +ç ÷ç ÷ç ÷è øè ø

Quadratic Assignment Problems-QAP

12 23 13AB BC ACCost f d f d f d= + +

Assign N Facilities to N locations Minimum sum of product of

“flow between facilities” and

“distance between locations”

N! Possible Solutions

Computationally hard problem, grows exponentially

N = 12, 479001600 solutions

N = 20, 2432902008176640000 solutions

How to find a solution from this ocean ?

Quadratic Assignment Problems-QAP

Problem Representation: ~ the distance between location k and l and ~ the traffic flow between facility i and j and

A solution: Permutation Matrix (NxN Matrix)

Quadratic Assignment Problems-QAP

[ ]klD d=

[ ]ijF f=

0 ,ii ij jif f f= =

[ ]ikP x=

0 ,kk kl lkd d d= =

1 if is assigned to 0 otherwise ik

i kx ì

= íî

Objective: Assign N Facilities to N locations Minimize the total cost of assignment

The constraints

Quadratic Assignment Problems-QAP

1 1 1 1

1min2

N N N N

ij kl ik jlx i j k lC f d x x

= = = =

= åååå

1

1

1 for 1,...,

1 for 1,...,

[0,1] ,

N

ikiN

ikk

ik

x i N

x k N

x i k

=

=

= =

= =

Î "

å

å

: Only one location k is assigned in each facility i

: Only one facility i is assigned in each location k

: Output level boundary

Quadratic function QAP

QAP Example

Neural Network as QAP solver

Find a representation for the problem

Define a problem energy function

Derive T and I matrixes from the energy function

Construct the network using T and I matrixes

Representation of QAP Permutation matrix represents an assignment Rows ~ Facilities Columns ~ Locations

12 23 34 45 51DA BC CB BE EDCost f d f d f d f d f d= + + + +

[ ]ikP x= =1 if is assigned to 0 otherwise ik

i kx ì

= íî

Representation of QAP Use a neuron to represent each entry of the matrix ��If the entry is 1, neuron is on ( ≈ 1) ��If the entry is 0, neuron is off ( ≈ 0)

N-facilities problem represented using N2 neurons

ikx

ikx

Hopfield Neural Network for QAP Arrange the neuron in a matrix form Neurons addressed with double indices

,1 1 1 1 1 1

1( )2

N N N N N N

ik jl ik jl ik iki j k l i k

E x T x x I x= = = = = =

= - -åååå åå

,1 1

( ) ( ) ( )N N

ikik ik jl jl ik

j l

du t u t T x t Idt = =

= - + +åå

0

( )1( ) ( ( )) 1 tanh2

ikik ik

u tx t f u ta

æ öæ ö= = +ç ÷ç ÷ç ÷è øè ø

Dynamic

Equation:

Output:

Energy

function:

Gaussian Machine for QAP

Same notation as Hopfield network

,1 1 1 1 1 1

1( )2

N N N N N N

ik jl ik jl ik iki j k l i k

E x T x x I x= = = = = =

= - -åååå åå

Dynamic

Equation:

Output:

Energy

function:

( )1( ) ( ( )) 1 tanh2 ( )

ikik ik

u tx t f u tæ öæ ö

= = +ç ÷ç ÷ç ÷è øè ø

,1 1

( ) ( ) ( ) ( )N N

ikik ik jl jl ik ik

j l

du t u t T x t I tdt = =

= - + + +åå

Neural Network as QAP solver

Find a representation for the problem

Define a problem energy function

Derive T and I matrixes from the energy function

Construct the network using T and I matrixes

Energy Function for QAP Its minima must correspond to the valid solutions Shorter paths and flow must have lower energy So, break it down into

penalty cost( )E x E E= +

Energy Function for QAP Constraint Satisfaction:

Cost:

2 2

penalty1 1 1 1 1 1

1 1 (1 )2 2 2

N N N N N N

ik ik ik iki k k i k i

A A CE x x x x= = = = = =

æ ö æ ö= - + - + -ç ÷ ç ÷

è ø è øå å å å åå

cost, 1 , 12

N N

ij kl ik jli j k l

BE f d x x= =

= åå

Only one “1” in each row Only one “1” in each column Output level close to “1”

Neural Network as QAP solver

Find a representation for the problem

Define a problem energy function

Derive T and I matrixes from the energy function

Construct the network using T and I matrixes

Mapping QAP onto Neural Network

Quadratic terms for T values Linear terms for I values

,, 1 , 1 , 1

1( )2

N N N

ik jl ik jl ik iki j k l i k

E x T x x I x= = =

= - -åå åQuadratic term Linear term

Mapping QAP onto Neural Network QAP Energy Function

Network Energy Function

2 2

1 1 1 1 1 1

, 1 , 1

( ) 1 1 (1 )2 2 2

2

N N N N N N

ik ik ik iki k k i k i

N N

ij kl ik jli j k l

A A CE x x x x x

B f d x x

= = = = = =

= =

æ ö æ ö= - + - + -ç ÷ ç ÷è ø è ø

+

å å å å åå

åå

,, 1 , 1 , 1

1( )2

N N N

ik jl ik jl ik iki j k l i k

E x T x x I x= = =

= - -åå å

, , 1 , , 1 , , , 1 , 1 , 1

1 1

( )2 2 2 2

2

N N N N N

ik jl ik jl ik jl ij kl ik jli k l k i j i j k l i j k l

N N

ik ik ikk i

A A C BE x x x x x x x f d x x

CA x A x x

= = = = =

= =

= + - +

- - -

å å å åå

å å

Mapping QAP onto Neural Network QAP Energy Function

Network Energy Function

,, 1 , 1 , 1

1( )2

N N N

ik jl ik jl ik iki j k l i k

E x T x x I x= = =

= - -åå å

Mapping QAP onto Neural Network

Network Energy Function

( )E x T x x I x= - -

Linear term

Mapping QAP onto Neural Network

Network Energy Function

Derived T and I matrices

( )E x T x x I x= - -

Linear term

, , , , ,

22

ik jl i j k l i j k l ij kl

ik

T A A C Bf dCI A

d d d d= - - + -

= +1 ,0 ,ik

i ji j

d=ì

= í ¹î

Kronecker Delta

Neural Network as QAP solver

Find a representation for the problem

Define a problem energy function

Derive T and I matrixes from the energy function

Construct the network using T and I matrixes

Agenda Hopfield Neural Network Gaussian Machine Quadratic Assignment Problems How to Solve Problem Computation Results Conclusion

Computation Results

Conclusion