MathWorks Technology Session at GE Optimization with ... · MathWorks Technology Session at GE...

1 © 2012 The MathWorks, Inc.

MathWorks Technology Session at GE

Optimization with MATLAB Products

November 8, 2012

2

MathWorks at a Glance

Earth’s topography

on a Miller cylindrical

projection, created

with MATLAB and

Mapping Toolbox

● Headquarters:

Natick, Massachusetts US

● Other U.S. Locations:

California, Michigan,

Texas, Washington, DC

● Europe:

France, Germany, Italy,

Netherlands, Spain, Sweden,

Switzerland, United Kingdom

● Asia-Pacific:

Australia, China, India,

Japan, Korea

● Worldwide training

and consulting

● Distributors serving more

than 20 countries

3

Optimization: What is it, Why use it?

Optimization in MATLAB

Gradient-based Optimization

Global Optimization

Additional Resources

Agenda

-2

0

2

-3-2

-10

12

3

-6

-4

-2

0

2

4

6

8

x

Peaks

y

Local minima

Global

minima

4

Optimization Problems

Minimize Risk

Maximize Profits

Maximize Fuel Efficiency

5

Design Process

Initial

Design

Variables

System

Modify

Design

Variables

Optimal

Design Objectives

met?

No

Yes

6

Why use Optimization?

Manually (trial-and-error or iteratively)

Initial

Guess

7


Automatically (using optimization techniques)

Initial

Guess

8


Finding better (optimal) designs

Faster Design Evaluations

Useful for trade-off analysis

Non-intuitive designs may be found

Antenna Design Using Genetic Algorithm http://ic.arc.nasa.gov/projects/esg/research/antenna.htm

http://ic.arc.nasa.gov/projects/esg/research/antenna.htm

9

Demos

Constrained optimization

Least squares / curve fitting

Quadratic programming

Finding global minimum

Mixed integer optimization

MATLAB Tips, Tricks, and Best

Practices along the way

1 75 150 2250

0.05

0.1

0.15

0.2

0.25

Assets

Fra

ction o

f in

vestm

ent

225-asset problem

Standard

With group constraints

Group 1 Group 2 Group 3

11

Optimization Approaches

Gradient Based Global Methods

Faster/fewer function

evaluations Larger problems (higher

dimensions) Finds local

minima/maxima

Finds global

minima/maxima (most of

the time)

Better on

non-smooth

stochastic

discontinuous

undefined gradients

Toolbox Optimization

Toolbox

Global

Optimization

Toolbox

12




Global Optimization


Agenda

-2

0

2

-3-2

-10

12

3

-6

-4

-2

0

2

4

6

8

x

Peaks

y

Local minima

Global

minima

13

Anatomy of an Optimization Problem

General Form for MATLAB Solvers

Objective Function

uxl

eqbxeqA

bAx

)(min xfx

Typically a linear or nonlinear function

Linear constraints

• inequalities

• equalities

• bounds

Nonlinear constraints

• inequalities

• equalities

Decision variables

0)(

0)(

xeq

c

xc

Subject to Constraints

14

Constrained Optimization Demo

min𝑥

log 1 + 𝑥1 −4

3

2

+ 3 𝑥1 + 𝑥2 − 𝑥13 2

15

Choosing the best solver

Doc: Optimization Toolbox > User’s Guide > Setting up an

Optimization > Choosing a Solver

Choose the algorithm based on your objective and constraint types.

16

Curve Fitting Demo

Given some data:

Fit a curve of the form:

teccty 21

)(

t = [0 .3 .8 1.1 1.6 2.3];

y = [.82 .72 .63 .60 .55 .50];

0 0.5 1 1.5 2 2.50.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

t

y

17

How to solve?

Ecc

cey

eccty

t

t

2

1

21

1

)(

As a linear system of equations:

2

2min yEc

Ecy

c

Can’t solve this exactly

(6 eqns, 2 unknowns)

An optimization problem!

2

1

3.2

6.1

1.1

8.0

3.0

0

1

1

1

1

1

1

50.0

55.0

60.0

63.0

72.0

82.0

c

c

e

e

e

e

e

e

18

Quadratic Programming

General Form Accepted by MATLAB Quadratic Solver

Objective Function

uxl

bxA

bAx

eqeq

xfHxxx

''min2

1

Quadratic function

Linear constraints

• inequalities

• equalities

• bounds

Decision variables

NOTE: In R2011a, quadprog was dramatically sped up for convex problems, particularly

significant for large ones. Uses sparse linear algebra, key for large, sparse problems.

Hessian (real, symmetric)

Subject to Constraints

19

Quadratic Programming Demo

1 75 150 2250

0.05

0.1

0.15

0.2

0.25

Assets

Fra

ction o

f in

vestm

ent

225-asset problem

Standard

With group constraints

Group 1 Group 2 Group 3

Given:

Historical data and a correlation matrix for 225 assets/

Find:

The portfolio weights that minimize risk.

C

w

Cwwrisk T

2

1

Portfolio weights

Covariance matrix

20

Gradient based Optimization Summary

Nonlinear minimization

– fminunc: unconstrained

– fmincon: linear and nonlinear constraints

• Supply Gradient and Hessian functions to

speed up

Constrained curve fitting with least squares solvers

– lsqlin: linear least-squares subject to linear constraints

– lsqnonlin: nonlinear least-squares with bound constraints

For linear or quadratic problems with linear constraints

– linprog: linear problems

– quadprog: quadratic problems

21




Global Optimization


Agenda

-2

0

2

-3-2

-10

12

3

-6

-4

-2

0

2

4

6

8

x

Peaks

y

Local minima

Global

minima

22

Why use Global Optimization?

Gradient based solvers are ineffective when the

gradient is undefined, and only converge to local

minima.

What if the problem is non-smooth, stochastic, has

multiple local minima?

Need a solver that can explore the solution space and

not fall into local minima.

*** No solver is guaranteed to find the global minimum

23

Global Optimization Solvers Covered Today

Multi Start

Global Search

Simulated Annealing

Pattern Search

Genetic Algorithm

24

Example Global Optimization Problems

Why does fmincon have a hard time finding the

function minimum?

0 5 10

-10

-5

0

5

10

x

Starting at 10

0 5 10

-10

-5

0

5

10

x

Starting at 8

0 5 10

-10

-5

0

5

10

x

Starting at 6

x s

in(x

) +

x c

os(2

x)

0 5 10

-10

-5

0

5

10

x

Starting at 3

0 5 10

-10

-5

0

5

10

x

Starting at 1

0 5 10

-10

-5

0

5

10

x

Starting at 0x s

in(x

) +

x c

os(2

x)

25


Why didn’t fminunc find the maximum efficiency?

Revolutions Per Minute, RPM

Manifold

Pre

ssure

Ratio

Peak VE Value = 0.96144

Start

End

1000 2000 3000 4000 5000 60000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

26


Why didn’t nonlinear regression find a good fit?

0 20 40 60 80 100 1200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

t

c

c=b1e-b

4t+b

2e-b

5t+b

3e-b

6t

27

Global Optimization

Goal:

Want to find the lowest/largest value of

the nonlinear function that has many local

minima/maxima

Problem:

Traditional solvers often return one of the

local minima (not the global)

Solution:

A solver that locates globally optimal

solutions Rastrigin’s Function

28

MultiStart Demo – Nonlinear Regression

lsqcurvefit solution MultiStart solution

29

MULTISTART

30

What is MultiStart?

Run a local solver from

each set of start points

Option to filter starting

points based on feasibility

Supports parallel

computing

31

MultiStart Demo – Peaks Function

32

GLOBAL SEARCH

33

What is GlobalSearch?

Multistart heuristic algorithm

Calls fmincon from

multiple start points to try

and find a global minimum

Filters/removes non-

promising start points

34

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview Schematic Problem

Peaks function

Three minima

Green, z = -0.065

Red, z= -3.05

Blue, z = -6.55

x

y

35

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 0 Run from specified x0

x

y

36

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1

3

6

0

0 0

4

0

-2

x

y

37

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

6

0

0 0

4

0

-2

x

y

38

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

39

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

40

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

41

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

42

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

43

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


6

Current penalty

threshold value : 4

x

y

44

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

45

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


Current penalty

threshold value : 4

-3

x

y

46

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2 Expand basin of attraction if minimum already found

Current penalty threshold value : 2

-0.1

x

y Basins can overlap

47

GlobalSearch Demo – Peaks Function

48

SIMULATED ANNEALING

49

What is Simulated Annealing?

A probabilistic metaheuristic

approach based upon the

physical process of annealing

in metallurgy.

Controlled cooling of a metal

allows atoms to realign from

a random higher energy state

to an ordered crystalline

(globally) lower energy state

50

Simulated Annealing Overview – Iteration 1 Run from specified x0

x

y

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

0.9

51

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Possible New Points:

Standard Normal N(0,1) * Temperature

Temperature = 1

52

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

11.01

1/)9.03(

Taccepte

P

53

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

54

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

55

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

56

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 0.75

0.3

-1.2

57

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N-1

3

x

y 0.9

Temperature = 0.1

0.3

-1.2

-3

58

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N Reannealing

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

59

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

27.01

1/))3(2(

Taccepte

P

60

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

61

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

62

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

63

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration …

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

-6.5

64

Simulated Annealing – Peaks Function

65

PATTERN SEARCH

(DIRECT SEARCH)

66

What is Pattern Search?

An approach that uses a

pattern of search directions

around the existing points

Expands/contracts around

the current point when a

solution is not found

Does not rely on gradients:

works on smooth and

nonsmooth problems

67

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1 Run from specified x0

x

y

3

68

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1 Apply pattern vector, poll new points for improvement

x

y

3

Mesh size = 1

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

0_*_ xvectorpatternsizemeshPnew

0]0,1[*1 x1.6

0.4

4.6

2.8

First poll successful

Complete Poll (not default)

69

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 2

x

y

3

Mesh size = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3 -2.8

Complete Poll

70

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

3

Mesh size = 4

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3 -2.8

71

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

3

Mesh size = 4*0.5 = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3 -2.8

72

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration N Continue expansion/contraction until convergence…

x

y

3 1.6

0.4

4.6

2.8

-4

0.3 -2.8

-6.5

73

Pattern Search – Peaks Function

74

GENETIC ALGORITHM

75

What is a Genetic Algorithm?

Uses concepts from

evolutionary biology

Start with an initial generation

of candidate solutions that are

tested against the objective

function

Subsequent generations

evolve from the 1st through

selection, crossover and

mutation

76

How Evolution Works – Binary Case

Selection

– Retain the best performing bit strings from one generation to the next.

Favor these for reproduction

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

Crossover

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

– child = [ 1 0 0 0 0 1 1 0 1 0 ]

Mutation

– parent = [ 1 0 1 0 0 1 1 0 0 0 ]

– child = [ 0 1 0 1 0 1 0 0 0 1 ]

77

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1 Evaluate initial population

x

y

78

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1 Select a few good solutions for reproduction

x

y

79

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2 Generate new population and evaluate

x

y

80

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2

x

y

81

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

82

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3


x

y

83

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration N Continue process until stopping criteria are met

x

y

Solution found

84

Genetic Algorithm – Peaks Function

85

Global Optimization Solvers

GlobalSearch, MultiStart

– Well suited for smooth objective and constraints

– Return the location of local and global minima

ga, simulannealbnd

– Many function evaluations to sample the search space

– Work on both smooth and nonsmooth problems

patternsearch

– Fewer function evaluations than ga, simulannealbnd

– Does not rely on gradient calculation like GlobalSearch and

MultiStart

– Works on both smooth and nonsmooth problems

86

Pattern Search Climbs Mount Washington

87

Genetic Algorithm – Integer Constraints

Mixed Integer Optimization

s.t. some x are integers

Examples

Only certain sizes of components

available

Can only purchase whole shares of stock

)(min xfx

88

Simulated Annealing – Eight Queens

89

Optimization and Parallel Computing

Many optimization algorithms involve some sort of parallelism – Gradient Estimation (fmincon)

– Evaluating each member of a Population (ga)

– Running a local solver from several start points (MultiStart)

Can use Parallel Computing Toolbox to distribute parallel tasks to

different workers

Demo: Parallel Computing with MultiStart

90

Speeding up with Parallel Computing

Global Optimization Algorithms that support Parallel Computing: – ga: Members of population evaluated in parallel at each iteration

– patternsearch: Poll points evaluated in parallel at each iteration

– MultiStart: Start points evaluated in parallel

Optimization Algorithms that support Parallel Computing: – fmincon: parallel evaluation of objective function for finite differences

– fminimax, fgoalattain: same as fmincon

Parallel Computing can also be used in the Objective Function – parfor

91

Parallel Computing Considerations

Need to consider structure of optimization problem when choosing

number of workers

Example – ga with default population size of 20

– Large speedup (~2x) from 1-2, 2-4 workers

– Smaller speedup (~1.25x) from 8-16 workers

1 worker 2 workers 4 workers 8 workers

Objective

Function

Evaluations

(1 generation)

92

Parallel computing required products

To use parallel computing To also use a cluster

Parallel Computing Toolbox MATLAB Distributed Computing Server

93

Scale Up to Clusters, Grids and Clouds

Desktop Computer

Parallel Computing Toolbox

Computer Cluster

MATLAB Distributed Computing Server

Scheduler

94

Summary of Global Optimization

Solvers designed to explore the solution space and find

global solutions

Problems can be stochastic, or nonsmooth

Solvers in Global Optimization Toolbox

– MultiStart

– GlobalSearch

– simulannealbnd

– patternsearch

– ga

Parallel Computing with ga, patternsearch, MultiStart

95




Global Optimization


Agenda

-2

0

2

-3-2

-10

12

3

-6

-4

-2

0

2

4

6

8

x

Peaks

y

Local minima

Global

minima

96

Learn More about Optimization with MATLAB

MATLAB Digest: Using Symbolic

Gradients for Optimization

Recorded webinar: Optimization

in MATLAB: An Introduction to

Quadratic Programming

Optimization Toolbox Web demo:

Finding an Optimal Path using MATLAB

and Optimization Toolbox

MATLAB Digest: Improving

Optimization Performance with

Parallel Computing

MathWorks Technology Session at GE Optimization with ... · MathWorks Technology Session at GE...

Documents

Transcript of MathWorks Technology Session at GE Optimization with ... · MathWorks Technology Session at GE...