Post on 04-Jan-2016
CPSC 5307
Search andDiscrete Optimization
Good results with problems that are too bigfor people or computers to solve completely
http://mathworld.wolfram.com/TravelingSalesmanProblem.html
Course outline
textbook – Michalewicz and Fogel(reasonable price, valuable book)
lectures, notes and pptppt presentations evaluation
assignments 5@10% 50% project 10% + 15% + 25% 50%
(documents plus presentations)
Difficult computing problems
hard to representwhat information, what data structurese.g., image organization
no known algorithmse.g., face recognition
no known efficient algorithmse.g., travelling salesperson
This course: discrete variable problems
Good results with problems too big to solve completely
cannot formally optimize results cannot be proven best usually results are not best
how to evaluate results? statistical probability
“within 4%, 19 times out of 20” satisficing
surpassing a threshhold that is “good enough”
Examples practical examples
scheduling (transportation, timetables,…) puzzles
crosswords, Sudoku, n Queens classic examples
SAT: propositional satisfiability problem(independent parameters)
CSP: constraint satisfaction problem(dependent parameters)
TSP: travelling salesman problem(permutations)
SAT: propositional satisfiability problem
P1 P2 P1^P2
F F F
F T T
T F F
T T T
n propositions, P1, P2, P3, …, Pn
What combination of truth values makes a sentence true?
Table has 2n rows.
n=50, 250 = 1,125,899,906,842,624
n=2; 22 = 4 rows
CSP: constraint satisfaction problem
example – map colouringn countries – 4 possible colours- constraints: adjacent
countries different colours- 4n combinations
n=13; 413 = 67,108,864 combinations; 25 constraints
TSP: traveling salesman(sic) problem
n cities: what is shortest path visiting all cities, C1, C2, C3, …, Cn once?
(n-1)! routes from home cityon complete graph
n = 16;(n-1)! = 1,307,674,368,000
C1
n = 5; (n-1)! = 24
Simple Example – one variable
maximize a function, f(n) over a range nmin ≤ n ≤ nmax
f (n) is linear f(n) is monotonic
f (n) is quadratic with f’’(n) < 0 f(n) is convex upward
f (n) is differentiable function f(n) has multiple (local) maxima
Simple Example – one variable
maximize a function, f(n) over a range nmin ≤ n ≤ nmax
f (n) is linear f(n) is monotonic
f (n) is quadratic with f’’(n) < 0 f(n) is convex upward
f (n) is differentiable function f(n) has multiple (local) maxima
Simple Example – one variable
maximize a function, f(n) over a range nmin ≤ n ≤ nmax
f (n) is linear f(n) is monotonic
f (n) is quadratic with f’’(n) < 0 f(n) is convex upward
f (n) is differentiable function f(n) has multiple (local) maxima
Problem description1. fitness function (optimization function, evaluation) –
e.g., m = n3 mod 1012. constraints (conditions) – e.g., n is oddfind global optimum of fitness function withoutviolating constraints ORgetting stuck at local optimum small space:
complete search large space: ?????
Large problems
more possible values more parameters, n = {n1, n2, n3, …} more constraints more complex fitness functions
- takes significant time to calculate m = f(n)
too big for exhaustive search
Searchingwithout searching everywhere
How to search intelligently/efficiently using information in the problem:-hill climbing-simulated annealing-constraint satisfaction-A*-genetic algorithms- …
Focusing search
assumption – some pattern to the distribution of the fitness function (otherwise: random distribution of fitness)
finding the height of land in a forest- can only see ‘local’ structure- easy to find a hilltop but are there other higher hills?
Fitness function distribution
convex – easy – start anywhere, make local decisions
Fitness function distribution many local maxima (conve in local region)
make local decisions but don’t get trapped
Performance – O(f(n))
What is the problem?
Quality software
Correctness Performance == Efficiency
Space – how much memory must be allocated?Time – how much time elapses during
execution?
Measuring performance
Testing Analysis of algorithms
How much space / time is required for an algorithm?
How does one algorithm compare to another?
How do requirements change depending on input?
Need a scale for describing performance
Measuring performance
“Big Oh” notation O(f(n))
Efficiency of performanceexpressed asan approximate function ofthe number of data elements processed
Three simple methodspublic void s0(int n){ int val = n;}
public void s1(int n){ int[] val = new int[n];}public void s2(int n){ int[][] val = new int[n][n];}
n = 6 Memory space
space
We will mainly considerTIME efficiency
Three simple methodspublic void m0(int n){ System.out.println(n);}
public void m1(int n){for(int i = 0; i<n; i++) System.out.println(n);}public void m2(int n){for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n);}
n = 6# of
executions
1
6
36
time
Three simple methods
public void m0(int n){ System.out.println(n);}
public void m0(int n){ System.out.println(n);}
public void m1(int n){ for(int i = 0; i<n; i++) System.out.println(n);}
public void m1(int n){ for(int i = 0; i<n; i++) System.out.println(n);}
n
time
n
time
n
time
public void m2(int n){ for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n);}
public void m2(int n){ for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n);}
O(1)
O(n)
O(n2)
Approximate measurepublic void m2(int n){ for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n);}public void m2a(int n){ for(int i = 0; i<n; i++) for(int j = i; j<n; j++) System.out.println(n);}
Better performance?
NO
Better performance?
NO
Approximate measure
1 2 6 10 100
# exe
c1 4 36 100 10000
# exe
c1 3 21 55 5050
public void m2(int n){ for(int i = 0; i<n; i++)
for(int j = 0; j<n; j++) System.out.println(n);}
public void m2a(int n){ for(int i = 0; i<n; i++)
for(int j = i; j<n; j++) System.out.println(n);}
n
n2
O(n2)
n2+n 2
O(n2)
Approximate measure
// linear search (unsorted array)public int search(double n, double[] arr){ int i = 0; while (i < arr.length && arr[i] != n) i++; return i;} From 1 to n iterations:
Failed search: n Average for successful search: n/2 Performance O(n)
BUT Different measure// binary search (sorted array)public int search(double n, double[] arr){int i, min = 0, max = arr.length-1; while (min <= max) { i = (min + max ) / 2; if (arr[i] == n) return i; if (arr[i] < n) min = i+1; else max = i-1; } return arr.length;}
From 1 to log n iterations: Failed search: log n Average for successful search: log n - 1 Performance O(log n)
O(log n) performance
n O(n)
n/2 O(n)
log2(n)O(log n)
O(log n) performance
examples: binary search in sorted arrays binary search tree operations
retrieve, update, insert, delete B-trees heaps
insert, remove
Sorting algorithms
two nested repeat operations both O(n) O(n2)
bubble, insertion, selection,… one O(n) and one O(log n) O(n log n)
mergesort, quicksort, heapsort,…
Shellsort: ??, O(n (log n)2), O(n1.25)
Analyzing an algorithm
rule of thumb: depth of nested structure
(or recursion) number of factors
in performance measure
some algorithms are difficult to analyze (e.g. hash table processing)
Comparing ‘polymomial’ performance
nfactors
O(1)0
O(log n)
1O(n)
1O(n log n)
2O(n2)
2O(n3)
3
2 1 1 2 2 4 8
8 1 3 8 24 64 512
32 1 5 64 320 1024 32768
128 1 7 128 896 16384 2097152
1024 1 10 1024 8192 1048576 1073741824
Combinatorial problems
Example: list the possible orderings of S:{1,2,3}: 123, 132, 213, 231, 312, 321
|S| 2 3 4 5 norderings 2 6 24 120 n!
Combinatorial problems
Example: generatetruth table for P^Q:
P Q P^Q
T T T
T F F
F T F
F F F
number of propositions 2 3 4 5 nrows in table 4 8 16 32 2n
Combinatorial problems
only known solutionshave n levels of nestingfor n data values
performance is exponential: O(en) includes n!, 2n, 10n,…
this is our kind of problem!
Comparing polymomial (P) andexponential (NP*) performance
nfactors
O(log n)
1O(n)
1O(n log n)
2O(n3)
3O(en), e.g. 2n
n
2 1 2 2 8 4
8 3 8 24 512 256
32 5 64 320 32768 4294967296
128 7 128 896 2097152 3.4028 x 1038
1024 10 1024 8192 1073741824 1.7977 x 10308
*Non-deterministic Polynomial
NP – Non-deterministic polynomial
if a solution is known, it can be verified in polynomial O(nk) timee.g., SATa solution (T,T,F,T,F,…,T,F,T,F,F,F)can be tested by substituting in the fitness function.
P and NP problems
P = NP?(Are there polynomial algorithms
for combinatorial problems?)YES there are but we are too stupid to find
themNO we need to find other ways to solve
these problems
(in 50+ years of trying, no answer to the question)
NP-hardNP-hard
P and NP problems
NPNP
PP
NP- c
om
ple
teco
mple
te
Representing problems
Abstractionfrom the messy real worldto the ordered simplicity of the model
Abstraction
Representing the problem for processing What information is relevant? How should it be operationalized?
Data Relations Processes
What information is irrelevant?
The puzzles
My real world problem
I have a list of 100 books to order online. Some are available in hardcover (H) or pocket (P) versions and some are available used(U)? How do I minimize my cost for the books?
Variables set of n controllable parameters
V = {x1, x2, x3, …, xn }
each variable xi has a set of possible values, domain Di
e.g., V = { x1, x2, ..., x100 }, 100 books to order
x1 ∈ D1 = {H,P,U}
x2 ∈ D2 = {H,P,U}
...x100 ∈ D100 = {H,P,U}
Problem Space
set of all possible combinations of variable values
dimension of spacen: number of variables
size of space: |D1|x|D2|x…x|Dn|
e.g., book ordersize = 3 x 3 x ... x 3 = 3100
sample point {H, H, U, ..., S}
Fitness
the objective function problem outcome as a function of
the variables: f(x1, x2, x3, …, xn) goal:
optimize (maximize or minimize) fe.g., total cost of purchase f(x1, x2, x3, …, x100)
minimize f
Constraints
rules C(V) that eliminate some points in the problem space from consideration
e.g., some books not available in all versions
Abstraction - “Operationalizing” representation of fitness and constraints for
evaluation and search
e.g. xi = vi = (type, cost, weight)i
fitness f=cost of books plus cost of shippingAll new books (H or S) are shipped in one order at $5 per
kilo of total weight but the shipping cost is waived if the total cost of the new books is over $100. Used books are shipped in a separate order with shipping cost of $3 per book.
How to handle constraints?
Abstraction - “Operationalizing”
e.g. xi = vi =(type, cost, weight)i
fitness f=cost of books plus cost of shippingAll new books (H or S) are shipped in one order at $5 per kilo of total
weight but the shipping rate is waived if the total cost of the new books is over $100. Used books are shipped in a separate order with shipping cost of $3 per book.
How to handle constraints?
cost = ∑ vi.cost // book cost
+ 3 x ∑ (vi.type == U) // ship used
+ [(∑ (vi.type<>U).vi.cost) < 100] // ship new
[5 x ∑ (vi.type<>U).vi.weight]
Exhaustive search
bestFitness = MaxVal (assume minimize)bestVfor all x1 ∈ D1, x2 ∈ D2, x3 ∈ D3,…xn ∈ Dn
if (x1, x2, x3,…xn satisfy constraints C(V))
fitness = f(x1, x2, x3,…xn)if (fitness < bestFitness)
bestFitness = fitnessbestV = {x1, x2, x3,…xn}
return bestFitness, bestV
Exhaustive search example
bestFitness = MaxValbestVfor all x1 ∈ D1, x2 ∈ D2, x3 ∈ D3,…x100 ∈ D100
if (x1, x2, x3,…x100 versions all exist)
cost = f(x1, x2, x3,…x100)if (cost < bestCost)
bestCost = costbestV = {x1, x2, x3,…x100}
return bestCost, bestV
Summary
Parameters, dimension, solution space Objective - fitness or evaluation function Constraints - impossible points in
solution space Finding point in space with optimal fitness
No algorithm to calculate point Space too large to search--> optimization methods with tradeoffs
Puzzles are Us
SEND+MOREMONEY
Each letter represents a different digitNo leading zero’sSum is correctFind assignment of digits to letters