Timo Berthold - Zuse Institute Berlin · 2020. 9. 12. · © 2019 Fair Isaac Corporation....

© 2019 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent. 1

© 2019 Fair Isaac Corporation. Confidential.

This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent.

Timo Berthold



• Fundamentals about Mathematical Optimization

• LP history

• Polyhedral theory

• LP theory


• Example: What is the shortest path from

• Zuse Institute Berlin• (where CO@Work should haven taken place)

• TU Berlin• (the university hosting CO@Work)

• Primal Bound: 8.07 km driving route

• Heuristic solution

• Addressing a different objective

• Dual Bound: Bee line: 6.74 km• https://www.distance.to/

• Gap: 8.07−6.74

8.07= 16.48%

image source: distance.to

https://www.distance.to/


• All representations can be converted into each other:

• min to max: multiply objective vector 𝑐 bei -1

• Equation to inequality: 𝑎𝑖𝑇𝑥 = 𝑏𝑖 → 𝑎𝑖

𝑇𝑥 ≤ 𝑏𝑖 , −𝑎𝑖𝑇𝑥 ≤ −𝑏𝑖

• ≤-inequality to ≥-inequality: multiply by -1

• Inequality to equation: Introduce slack variable, 𝑎𝑖𝑇𝑥 ≤ 𝑏𝑖 → 𝑎𝑖

𝑇𝑥 + 𝑠𝑖 = 𝑏𝑖

• Unbounded variable to bounded: 𝑥 = 𝑥+ − 𝑥−, 𝑥 ∈ ℝ, 𝑥+, 𝑥− ∈ ℝ≥0

• Bounded to unbounded: Consider bounds as constraints

• LP literature typically uses the standard form max {𝑐𝑇𝑥 | 𝐴𝑥 = 𝑏, 𝑥 ≥ 0}

• MIP literature often uses inequalities for the constraints


• Egyptians and Babylonians considered about 2000 B.C. the solution of special linear equations. They described examples and did not formulate methods in today’s style.

• What we call Gaussian elimination today has been explicitly described in Chinese “Nine Books of Arithmetic” which is a compendium written in the period 2010 B.C. to A.D. 9, but the methods were probably known even before that.

• Gauss, by the way, never described Gaussian elimination. He just used it and stated that the linear equations he used can be solved per eliminationem vulgarem


• In 1827 Fourier described a variable elimination method for linear inequalities, today often called Fourier-Motzkin elimination (Motzkin 1936).

• By adding one variable and one inequality, Fourier-Motzkin elimination can be turned into an LP solver.

• Who formulated the first LP?

• The usual credit goes to George J. Stigler (1939)

• Full example:

• 77 foods, 9 nutritients

• Stigler’s heuristic solution was 0.7% from optimal


• 1939 L. V. Kantorovitch (1912-1986): Foundations of linear programming

• 1947 G. B. Dantzig (1914-2005): Invention of the (primal) simplex algorithm

• 1954 C.E. Lemke & E.M.L. Beale: Dual simplex algorithm

• 1953 G.B. Dantzig, 1954 W. Orchard Hays, and 1954 G. B. Dantzig & W. Orchard Hays: Revised simplex algorithm


• The first commercial LP-Code was on the market in 1954 and available on an IBM CPC (card programmable calculator)

• Record: 71 variables, 26 constraints, 8 h running time

• About 1960: LP became commercially viable, used largely by oil companies

• 1972: first commercial IP solver (almost 50 years ago)


• Leonid V. Kantorovich and Tjalling C. Koopmans received the Nobel Prize for Economics in 1975 for their work on „Optimal use of scarce resources“ – essentially for thefoundation and economic interpretation of LP

image source: Wikipedia


1. Dantzig‘s simplex algorithm was published in:

a) 1947

b) 1954

c) 1958

2. The set of feasible solutions for a MIP is also called:

a) a simplex set

b) a mixed integer set

c) a linear set

3. Who formulated the diet problem?

a) George Stigler

b) Leonid V. Kantorovich

c) Jean B. J. Fourier


For nice, interactive visualiza-tions of the 120 regular convex polyhedra, check out:https://polyhedra.tessera.li/


https://polyhedra.tessera.li/


• Linear programming is optimizing over an n-dimensional variable vector

• polyhedron: intersection of finitely many halfspaces

• P = {𝑥 ∈ ℝ𝑛 | 𝐴𝑥 ≤ 𝑏}

• polytope: convex hull of finitely many points

• P = conv(V), V a finite set in ℝ𝑛.

• convex polyhedral cone: conic combination (i.e., nonnegative linear combination) of finitely many points

• K = cone(E), E a finite set in ℝ𝑛.



image source: stackexchange.com


• Theorem: For a subset P of ℝ𝑛 the following are equivalent:

1. P is a polyhedron.

2. P is the intersection of finitely many halfspaces, i.e., there exist a matrix A and a vector b with P = {𝑥 ∈ ℝ𝑛 | 𝐴𝑥 ≤ 𝑏} (outer representation)

3. P is the sum of a convex polytope and a finitely generated (polyhedral) cone, i.e., there exist finite sets V and E with P = conv(V) + cone(E) (inner representation)


• 𝑃 = 𝑐𝑜𝑛𝑣 0, {𝑒𝑖 | 𝑖 ∈ 1,… , 𝑛 𝑃 = 𝑥 𝑥 ≥ 0,σ𝑖=1𝑛 𝑥𝑖 ≤ 1}

• Simplex: n+1 points, n+1 inequalities

• 𝑃 = 𝑐𝑜𝑛𝑣 {−𝑒𝑖 , 𝑒𝑖 | 𝑖 ∈ {1,… , 𝑛} 𝑃 = 𝑥 𝑎𝑇𝑥 ≤ 1, ∀ 𝑎 ∈ {−1,1}𝑛}

• Cross polytope: 2n points, 2n inequalities

• 𝑃 = 𝑐𝑜𝑛𝑣 {−1,1}𝑛 𝑃 = 𝑥 𝑎𝑇𝑥 ≤ 1, ∀ 𝑎 ∈ {−𝑒𝑖 , 𝑒𝑖}, 𝑖 ∈ {1, … , 𝑛}}

• Cube: 2n points, 2n inequalities

image source: aliexpress.com

image source: healingcrystals.com

image source: naturshop.cz


An n-dimensional polyhedron has the following different faces:

• Vertex (0-dimensional)

• Edge (1-dimensional)

• ...

• Ridge = subfacet (n-2)-dimensional

• Facet (n-1)-dimensional

and each of them is a polyhedron itself!

image source: Carmelly Griffith


• The Farkas-Lemma (1908):

• A polyhedron defined by an inequality system 𝐴𝑥 ≤ 𝑏 is empty, if and only if there is a vector y such that 𝑦 ≥ 0, 𝑦𝑇𝐴 = 0𝑇 , 𝑦𝑇𝑏 < 0

• Hence, we can reformulate 𝐴𝑥 ≤ 𝑏 to a wrong statement

• Theorem of Alternatives

• Foundation of all important higher-level LP theory: Duality theorems, complementary slackness, proof of LP optimality, etc.

• Example: 𝑥 ≤ 1,−𝑥 ≤ −2 ⇒ 𝐴 =1−1

, b =1−2

. Choose y =11

.

• 𝑦𝑇𝐴 = 1 ⋅ 1 + 1 ⋅ −1 = 0 and 𝑦𝑇𝑏 = 1 ⋅ 1 + 1 ⋅ −2 = −1 < 0

• We added the two inequalities and got a wrong statement


• Fourier, 1827, rediscovered by Motzkin, 1936

• Method: successive projection of a polyhedron in n-dimensional space into a vector space of dimension n-1 by elimination of one variable.

• Can check whether a polyhedron is nonempty

• Can be used to prove Farkas Lemma

• Can be used for Linear Programming


• Each elimination step might square the number of rows, hence in total 𝑂(𝑚2𝑛)

• Fourier-Motzkin essentially the best-known method for polyhedral transformations:

• Let a polyhedron be given via: P = conv(V)+cone(E)

• Goal: Find a representation of P in the form P = {𝑥 ∈ ℝ𝑛 | 𝐴𝑥 ≤ 𝑏}

• Idea: Write P as follows : P = {𝑥, 𝑦, 𝑧 ∈ ℝ𝑑| 𝑥 = 𝑉𝑦 + 𝐸𝑧,σ 𝑦𝑖 = 1, 𝑦 ≥ 0, 𝑧 ≥ 0}

• Eliminate y and z

• With some tricks, FME can be reduced to single-exponential running time

• which is already best possible for cube/cross polytope


• How do we get a proof of solution quality?

• Easy case: One of our constraints underestimates the objective function

• E.g., if 𝑐𝑇𝑥 = 2𝑥1 + 𝑥2 and one constraint 𝑥1 + 𝑥2 ≥ 3, then also min 𝑐𝑇𝑥 ≥ 3

• Observation 1: Constraints are invariant to scaling by positive numbers

• Observation 2: The sum of two valid constraints is a valid constraint

• Consequence: Conic combinations of constraints are valid

• Task: Find a conic combination of all constraints that is a maximal underestimator for our objective

• This is an LP. The dual LP

min{𝑐𝑇𝑥 | 𝐴𝑥 ≥ 𝑏, 𝑥 ≥ 0}



• Conic combination: 𝑦 ≥ 0





• combination of all constraints...: yT𝐴𝑥 ≥ 𝑦𝑇b






• ...that is an (...)underestimator for our objective: y𝑇𝐴 ≤ 𝑐







• ...and that is a maximal underestimator ∶ max 𝑦𝑇𝑏







• ...and that is a maximal underestimator ∶ max 𝑦𝑇𝑏

• This is an LP. The dual LP!

max{𝑦𝑇𝑏| yT𝐴 ≤ 𝑐, 𝑦 ≥ 0}



primal: min{𝑐𝑇𝑥 | 𝐴𝑥 ≥ 𝑏, 𝑥 ≥ 0} and dual: max{𝑦𝑇𝑏| yT𝐴 ≤ 𝑐, 𝑦 ≥ 0}

• Each constraint in the primal LP becomes a variable in the dual LP

• Each variable in the primal LP becomes a constraint in the dual LP

• The objective direction is inversed – maximize in the primal becomes minimize in the dual and vice-versa

• Inequality constraints become ≥-bounds

• Consequently, equations become free variables

By this scheme: Easy to see that the dual of the dual is the primal


max{𝑦𝑇𝑏| 𝑦𝑇𝐴 ≤ 𝑐, 𝑦 ≥ 0} ≤ min{𝑐𝑇𝑥 | 𝐴𝑥 ≥ 𝑏, 𝑥 ≥ 0}

• Trivial to proof:

• 𝑦𝑇𝑏 ≤ 𝑦𝑇 𝐴𝑥 = (𝑦𝑇𝐴)𝑥 ≤ 𝑐𝑇𝑥

• q.e.d.


max{𝑦𝑇𝑏| 𝑦𝑇𝐴 ≤ 𝑐, 𝑦 ≥ 0} ≤ min{𝑐𝑇𝑥 | 𝐴𝑥 ≥ 𝑏, 𝑥 ≥ 0}

• Trivial to proof:

• 𝑦𝑇𝑏 ≤ 𝑦𝑇 𝐴𝑥 = (𝑦𝑇𝐴)𝑥 ≤ 𝑐𝑇𝑥

• q.e.d.

image source: Twitter


• The most important and influential theorem in Optimization

• Primal has a finite optimum if and only if dual has a finite optimum

• min{𝑐𝑇𝑥 | 𝐴𝑥 ≥ 𝑏, 𝑥 ≥ 0} = max{𝑦𝑇𝑏| yT𝐴 ≤ 𝑐, 𝑦 ≥ 0}

• A relation of this type is called min-max result

• Proof is not straight-forward, uses weak duality and Farkas lemma

• Three possibilities: finite (and equal) optima, unbounded and infeasible, both infeasible


• min{𝑐𝑇𝑥 | 𝐴𝑥 ≥ 𝑏, 𝑥 ≥ 0} = max{𝑦𝑇𝑏| y𝑇𝐴 ≤ 𝑐, 𝑦 ≥ 0}

• At an optimal solution pair (x,y), for all i: either 𝑦𝑖 = 0 or σ𝑗=1𝑛 𝑎𝑖𝑗 𝑥𝑗 = 𝑏𝑖 (or both)

• Analogously, for all j either 𝑥𝑗 = 0 or σ𝑗=1𝑛 𝑎𝑖𝑗 𝑦𝑖 = 𝑐𝑗

• Proof: By weak duality

• „to construct an optimality proof, we can only use constraints that are tight at the optimal point“

• Interpretation of dual variables as shadow prices: How much would the objective increase, if we relaxed the constraint?

• There is a solution, s.t. 𝑦𝑖 > 0 ⇔ σ𝑗=1𝑛 𝑎𝑖𝑗 𝑥𝑗 = 𝑏𝑖 (strong complementary slackness)


• naïvely: one optimal solution, determined by n constraints

• at a second thought: one optimal solution, k > n tight constraints

• but really: an optimal polyhedron

• Can be exploited in some ways. But mostly a burden


1. If a dual variable has a nonzero value in an optimal primal-dual solution:

a) the corresponding primal variable is nonzero as well

b) the corresponding primal constraint is nonzero as well

c) the corresponding primal constraint is tight

2. Describing a polyhedron as P = {𝑥 ∈ ℝ𝑛 | 𝐴𝑥 ≤ 𝑏} is called the

a) Standard form

b) Outer representation

c) Inner representation

3. The dual of min{𝑐𝑇𝑥 | 𝐴𝑥 ≥ 𝑏, 𝑥 ≥ 0} is

a) min{𝑦𝑇𝑏| y𝑇𝐴 = 𝑐, 𝑦 ≤ 0}

b) max{𝑦𝑇𝑏| y𝑇𝐴 = 𝑐, 𝑦 ≥ 0}

c) max{𝑦𝑇𝑏| yT𝐴 ≤ 𝑐, 𝑦 ≥ 0}


1. If a dual variable has a nonzero value in an optimal primal-dual solution:

a) the corresponding primal variable is nonzero as well

b) the corresponding primal constraint is nonzero as well

c) the corresponding primal constraint is tight

2. Describing a polyhedron as P = {𝑥 ∈ ℝ𝑛 | 𝐴𝑥 ≤ 𝑏} is called the

a) Standard form

b) Outer representation

c) Inner representation

3. The dual of min{𝑐𝑇𝑥 | 𝐴𝑥 ≥ 𝑏, 𝑥 ≥ 0} is

a) min{𝑦𝑇𝑏| y𝑇𝐴 = 𝑐, 𝑦 ≤ 0}

b) max{𝑦𝑇𝑏| y𝑇𝐴 = 𝑐, 𝑦 ≥ 0}

c) max{𝑦𝑇𝑏| y𝐓𝐴 ≤ 𝑐, 𝑦 ≥ 0}


© 2019 Fair Isaac Corporation. Confidential.

This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent.

Timo Berthold

Timo Berthold - Zuse Institute Berlin · 2020. 9. 12. · © 2019 Fair Isaac Corporation....

Documents

Transcript of Timo Berthold - Zuse Institute Berlin · 2020. 9. 12. · © 2019 Fair Isaac Corporation....