Engineering Optimization and Metaheuristics Swarm Optimization Firefly Algorithm & Cuckoo Search...

Xin-She Yang 1 / 40

Engineering Optimization and Engineering Optimization and MetaheuristicsMetaheuristics

Xin-She Yang @ 2010

Xin-She Yang 2 / 40

MetaheuristicsMetaheuristics and Engineering Optimizationand Engineering Optimization

� Introduction

� Design Optimization

� Conventional Approach

� Metaheuristics

� Genetic Algorithms & Simulated Annealing

� Particle Swarm Optimization

� Firefly Algorithm & Cuckoo Search

� Examples

� Conclusions

Xin-She Yang 3 / 40

Part I: Introduction to Optimization Part I: Introduction to Optimization

� Introduction

� Function optimization

� KKT conditions

Xin-She Yang 4 / 40

RosenbrockRosenbrock’’ss Banana FunctionBanana Function2 2 2f (x, y) (1 x) 100 (y x )= − + − fmin=0 at (1,1).

-2

-1

0

1

2

-2

-1

0

1

20

1000

2000

3000

4000

-2

-1

0

1

2

-2

-1

0

1

20

2

4

6

8

10

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

500

1000

1500

2000

2500

3000

Xin-She Yang 5 / 40

KarushKarush--KuhnKuhn--Tucker (KKT) ConditionsTucker (KKT) Conditions

nMinimize f ( )

Subject to

∈ℜxx

i

i

h ( ) 0, (i 1, 2,...,M)

g ( ) 0, ( j 1, 2,..., N)

= =

≤ =

x

x

M N

0 * i i * j *

i 1 j 1

f ( ) h ( ) g ( ) 0,= =

µ ∇ + λ ∇ + ∇ =∑ ∑x x x

If all the functions are continuously differentiable, at a local minimum,

there exist such that1 M 0 1 Nan,..., , ..d ,. ,λ λ µ µ µ

j * j j *

j

g ( ) 0, g ( ) 0, ( j 1, 2,..., N),

0, ( j 1, 2,..., N).

≤ µ = =

µ ≥ =

x x

Xin-She Yang 6 / 40

Part II: Optimization Algorithm and Part II: Optimization Algorithm and MetaheuristicsMetaheuristics

� Type of Optimization Algorithms

� Conventional Algorithms

� Metaheuristic Algorithms

(Bio-inspired or nature-inspired)

Xin-She Yang 7 / 40

Optimization AlgorithmsOptimization Algorithms

Deterministic:

Newton’s (1669, published in 1711); Newton-Raphson (1690); Hill-

climbing/steepest descent (Cauchy 1847); Least squares (Gauss

1795); Linear programming (Dantzig 1947), conjugate gradient

(Lanczos et al 1952); Interior-point method (Karmarkar 1984), etc

Stochastic: Genetic algorithms, random search,

heuristic/metaheuristic algorithms, etc.

Algorithms:

deterministic & stochastic

Xin-She Yang 8 / 40

MetaheuristicMetaheuristic AlgorithmsAlgorithms

Heuristic search (Turing 1940s); Genetic algorithms (Holland

1960s/1970s); Evolution strategy (Rechenberg & Swefel 1960s);

Evolutionary programming (Fogel et al. 1960s);

Simulated annealing (Kirkpatrick et al. 1983); Tabu search

(Glover 1980s); Ant colony optimization (Dorigo 1992); Genetic

programming (Koza 1992); Particle swarm optimization (Kennedy

& Eberhart 1995); Differential evolution (Storn & Price 1996/97);

Harmony search (Geem 2001); Honeybee algorithm (Nakrani &

Tovey 2004);…… Firefly algorithm (Yang 2008); Cuckoo search

(Yang & Deb 2009);

meta- = higher-level, beyond heuristic = by trial and error

Xin-She Yang 9 / 40

GradientGradient--based Methodsbased Methods

n 1 n

f,+

∇= −

Hx x

For a function f(x) in n-dimensions where x=(x1,…, xn),

Newton’s method can be written as

where H is the Hessian matrix

2 2

2

1 1 n

n

2 2

2

n 1 n

f f...

x x x

H( ) : :

f f...

x x x

∂ ∂

∂ ∂ ∂ =

∂ ∂ ∂ ∂ ∂

x

Xin-She Yang 10 / 40

Steepest Descent and HillSteepest Descent and Hill--ClimbingClimbing

n 1 n nf ( )+ = − α ∇Ix x x

If H-1 is replaced by I, it becomes the

quasi-Newton or steepest descent method

where α is the step size.

The actual move is along the negative gradient direction,

and α can vary during iterations.

n n( ) f ( )= −α ∇s x x


Search for OptimalitySearch for Optimality

n n2

i i

i 1 i 1

1 1 1f ( ) 20exp x exp cos(2 x ) 20 e

5 n n[ ] [ ]

= =

= − − − π + +∑ ∑x

Ackley’s function

which has a global minimum f*=0 at (0,0,…,0) for all

-32.768 ≤ xi ≤ 32.768.


Search Methods: Potential DifficultiesSearch Methods: Potential Difficulties


Random Restart Hill-climbing


Genetic Algorithms

(J. Holland 1970s)

x=(x1, x2, …, xd)

1 0 0 1 0 1 0 0 1 0 0 1 0 1 1 1

1 1 1 1 0 1 0 1

1 0 0 1 0 1 0 0

0 1 1 0 0 1 0 1

0 1 1 0 0 1 0 1

1 1 1 1 0 1 0 1 1 0 0 1 0 1 1 1

parents

children

crossover

1 0 0 1 0 1 0 0

mutation

1 0 0 0 0 1 0 0


Genetic Algorithms Genetic Algorithms


Genetic Algorithms Genetic Algorithms

Generation t=0

Generation t=20

Generation t=50


exp[ ],E

pkT

= −Simulated Annealing

T , p 1. → ∞ →

0, 0,T p→ → Only good moves are accepted

(Hill - climbing)


Swarm Intelligence

BBC Swarm Intelligence

Birds – Starlings over Rome (Video)


Particle Swarm Optimization (PSO)

Kennedy et al. (1995)

1 * *

1 2

1 1

( ) ( ),t t t t

i i i i i

t t t

i i i

v v g x x x

x x v

αε βε+

+ +

= + − + −

= +

xi

g*


PSO Convergence PSO Convergence ((ClercClerc and Kennedy 2002and Kennedy 2002))

1

1 2

1 1

( ) ( )t t t t

i i i i i

t t t

i i i

v v x p x g

x x v

ε ε+

+ +

= + − + −

= +

1 2

1 2

ip g

pε ε

ε ε

+=

+Let

We have a dynamical system

1 2 ,t t

y p xφ ε ε= + = −

1

1(1 )

t t t

t t t

v v y

y v y

φ

φ+

+

= +

= − + −

1

1, ,

1 1

t

t t t

t

vU A U AU

y

φ

φ +

= = ⇒ = − −

Eigenvalues:

2 41

2 2

φ φφλ

−= − ±

Periodic, quasi-periodic

Convergence around 4φ =

individual best global


PSO Examples: Michalewicz’s function

22( ) sin( ) sin ( )m iix

fπ=

= −∑d

i 1

ix x


PSO Examples


PSO –Very Efficient

but may have problems (sometimes)


Firefly Algorithm (FA) by Xin-She Yang (2008)

Firefly Algorithm

• Fireflies are unisex and brightness/attractiveness varies

with distance

• Less bright ones will be attracted to brighter ones

• If no bright firefly can be seen, it moves randomly

1 2

0 exp[ ]( )t t t t

i i ij j ix x r x x Levyβ γ α+ = + − − + ⊗


Firefly Algorithm (Firefly Algorithm (pseudo code)pseudo code)

Objective f(x)

Initialize a population of fireflies xi (i=1, …, n)

Define parameters (e.g., γ) and formulate light intensity I ~ f(x)

While (stop criteria)

for i=1:n (all fireflies)

for j=1:n (all fireflies)

if Ij>Ii, move firefly i toward j; end if

Vary attractiveness/light via exp(- γ r2);

Evaluate new solutions and update;

end (for j)

end (for i)

Rank fireflies and find the current best;

End while

Post-processing and visualization


FA Examples2

1

( ) ( | | ) exp[ sin( )]d

i i

i

f x x= =

= −∑ ∑d

i 1

x


Cuckoo Birds

– not just beautiful with amazing sound …

-- Breeding behaviour (Obligate brood parasitism)

(BBC Wildlife Video)


Cuckoo Search (by Xin-She Yang and Suash Deb, 2009)

� Each cuckoo lays one egg at a time

and dumps it in a randomly-chosen nest (solution)

� Best nests/eggs (quality solutions) will carry over to

the next generation

� Bad eggs (not-so-good solutions) will be discovered and

thrown away with a probability p, and will be replaced by

a new solution (randomly generated)


Cuckoo Search (pseudo code)Cuckoo Search (pseudo code)

Objective f(x) (in d dimensions)

Generate an initial population of n host nests;

While (stop criterion)

Get a cuckoo randomly (say, i);

Replace its solution by performing a random walk/Levy flight

Evaluate its fitness (Fi)

Choose a nest (say j among n) randomly;

If Fi > Fj. Replace j by the new solution; end if

A fraction (pa) of nests are abandoned/replaced by new solutions;

Rank the solutions and find the current best;

Keep the best solutions (pass onto the next generation);

End while

Post-processing and visualization.


Cuckoo Search Examples: Rosenbrock’s function

2 2 2f (x, y) (1 x) 100 (y x )= − + −


Part III: Applications (Examples) Part III: Applications (Examples)

� Three-bar truss design

� Pressure vessel design

� Speed reducer (gearbox)


Function OptimizationFunction OptimizationWe usually use about 30 to 50 standard test functions for validation.

For example, Michalewicz’s function

In 2D, fmin= -1.80 at (2.20, 1.57)

2nj2m

j

j 1

jxf ( ) sin(x )sin ( ), (m 10).

=

= − =π

∑x


ThreeThree--Bar Truss Design Bar Truss Design

1 20 A 1, 0 A 1,

H 100 cm,

P 20 MPa, 20 MPa.

≤ ≤ ≤ ≤

=

= σ =

( )1 2 1 2f (A , A ) 2 2A A H= +

1 21 2

1 1 2

2A Ag P 0

2A 2A A

+= − σ ≤

+

22 2

1 1 2

Ag P 0

2A 2A A= − σ ≤

+ 3

1 2

1g P 0

A 2A= − σ ≤

+

To minimize weight/materials

Subject to


Optimal solutions (truss design example)Optimal solutions (truss design example)

Ray and Saini (2001) Tsai (2005) Present Study (CS)

A1

0.795 0.788 0.78867

A2

0.395 0.408 0.40902

g1

-0.00169 0.000818 -0.00029

g2

-0.26124 -0.2674 -0.26853

g3

-0.74045 -0.73178 -0.73176

fmin

264.3 263.68 263.9716


Pressure Vessel Design Pressure Vessel Design

Minimize

Subject to:

where 1×0.0625 ≤ Ts, Th ≤ 99×0.0625 and 10 ≤ R, L ≤ 200

1 sg T 0.0193 R 0= − + ≤ 2 hg T 0.0095 R 0= − + ≤

2 2 3

3

4g R L R 1, 296,000 0

3= −π − π + ≤

4g L 240 0,= − ≤

2 2 2

s h s h s hf (T ,T ,R,L) 0.6224 T R L 1.7781T R 3.1661T L 19.84 T L= + + +



Speed Reducer Design OptimizationSpeed Reducer Design Optimization2 2

1 2 3 3

2 2 3 3 2 2

1 6 7 6 7 4 6 5 7

Minimize f ( ) 0.7854 x x (3.3333x 14.9334x 43.0934)

1.508x (x x ) 7.4777(x x ) 0.7854(x x x x )

= + −

− + + + + +

x

1 2

1 2 3

27Subject to g ( ) 1 0

x x x= − ≤x

2 2 2

1 2 3

397.5g ( ) 1 0

x x x= − ≤x

3

43 4

2 3 6

1.93xg ( ) 1 0

x x x= − ≤x

3

54 4

2 3 7

1.93xg ( ) 1 0

x x x= − ≤x

2 645 3

6 2 3

745.0x1.0g ( ) 16.9 10 1 0

110x x x( )= + × − ≤x 2 65

6 3

7 2 3

745.0x1.0g ( ) 157.5 10 1 0

85x x x( )= + × − ≤x

28

1

5xg ( ) 1 0

x= − ≤x 1

9

2

xg ( ) 1 0

12x= − ≤x

2 37

x xg ( ) 1 0

40= − ≤x

610

4

1.5x 1.9g ( ) 1 0

x

+= − ≤x 7

11

5

1.1x 1.9g ( ) 1 0

x

+= − ≤x

1 2 3 4

5 6 7

2.6 x 3.6, 0.7 x 0.8, 17 x 28, 7.3 x 8.3,

7.8 x 8.4, 2.9 x 3.9, 5.0 x 5.5

≤ ≤ ≤ ≤ ≤ ≤ ≤ ≤

≤ ≤ ≤ ≤ ≤ ≤


Speed Reducer DesignSpeed Reducer Design

fmin=2,996.348165

Best optimal solution (Cagnina et al. 2008)

Our new best (2010)

*(3.5,0.7,17,7.3,7.8,3.34336449,5.285351)=x

fmin=2,993.7495888

Work in progress: design problems with about 1000 variables

(300 discrete variables,700 continuous variables,

and about 4000 nonlinear constraints).


Open ProblemsOpen Problems

No convergence analysis of most metaheuristic algorithms

(preliminary studies exist for SA and PSO)

Stopping criteria:

when to stop, if global optimality is reached/reachable

Guidelines of choice of parameters:

balance of intensification and diversification

Measure of performance and what to compare ?

(no agreed guidelines)

No-free-lunch theorems (Wolpert and Macready, 1997)

for multiobjective optimization ?


Yang X. S., Nature-Inspired Metaheuristic Algorithms, Luniver Press, (2008)

Yang X. S., Introductory Mathematics for Earth Scientists, Dunedin Academic, (2009).

Yang X. S., Introduction to Computational Mathematics, World Scientific, (2008)

Yang X. S., Engineering Optimisation: An Introduction with Metaheuristic Applications, Wiley (2010)(2010)

Int. J. of Mathematical Modelling and Numeric Optimisation (IJMMNO)

http://www.inderscience.com/ijmmno

Thank you very much ☺

Engineering Optimization and Metaheuristics Swarm Optimization Firefly Algorithm & Cuckoo Search...

Documents

Transcript of Engineering Optimization and Metaheuristics Swarm Optimization Firefly Algorithm & Cuckoo Search...