Nonconvex Quadratic Problems and Games with SeparableConstraints
Javier Zazo Ruiz
Universidad Politecnica de Madrid
14 of December 2017
ETSITESCUELA TECNICA SUPERIOR DE INGENIEROS DE TELECOMUNICACIÓN
Outline
1 Nonconvex QPs with separable constraints
2 Squared ranged localization problem
3 Algorithmic framework of QPs
4 Other works
5 Concluding remarks
Outline
1 Nonconvex QPs with separable constraintsIntroduction to QPsRoadmap to establish strong duality in QPsRobust least squares with multiple constraints
2 Squared ranged localization problem
3 Algorithmic framework of QPs
4 Other works
5 Concluding remarks
Quadratically constrained quadratic problem (QP)
I Let’s consider a general QP:
minx∈Rp
xTA0x + 2bT0 x + c0
s.t. xTAix + 2bTi x + ci ≤ 0 ∀i = 1, . . . , N.
where A0, Ai are symmetric matrices and b0, bi, x ∈ Rp, c0, ci ∈ R.
I If A0 � 0 and every Ai � 0 the problem is convex (≈ easy to solve).
I Otherwise, the problem is non-convex (local minima may exist).
I These problems are generally NP-Hard.
I Use of QPs is vast.
Javier Zazo Nonconvex QPs and games 1 / 28
Polynomial minimizationI Minimize a polynomial over a set of of polynomial inequalities:
min p0(x)
s.t. pi(x) ≤ 0, i = 1, . . . ,m.
I Rename variables and add them as constraints.
I Example:
minx,y,z
x3 − 2xyz + y + 2
s.t. x2 + y2 + z2 − 1 = 0.
Introducing change of variables u = x2, v = yz,
min ux− 2vx+ y + 2
s.t. x2 + y2 + z2 − 1 = 0
u− x2 = 0
v − yz = 0.
Javier Zazo Nonconvex QPs and games 2 / 28
Polynomial minimizationSix-hump-camel problem
−2−1
01
2
−1−0.5
00.5
1
0
5
x1x2
f(x 1
,x2)
Javier Zazo Nonconvex QPs and games 3 / 28
Partinioning problemsAlso called “Boolean Optimization”
minx
xTA0x
s.t. xi ∈ {−1, 1}
I The problem is NP-hard (even if A0 � 0).
I Binary constraints xi ∈ {−1, 1} ⇐⇒ x2i = 1.
I The MAXCUT � benchmark problem.
Javier Zazo Nonconvex QPs and games 4 / 28
Transmit beamforming problem
I Determine optimal beams for downlink transmissions.
I The beamformers affect the system performance, causing interference.
minwi,∀i
∑i∈N
wHi wi
s.t. SINRi(wi, w−i) ≥ Γi
I The above problem can be relaxed:
minWi,∀i
∑i∈N
tr[Wi]
s.t. SINRi(Wi,W−i) ≥ Γi
Wi = WiH
Wi � 0 ∀i ∈ N
Mats Bengtsson and Bjorn Ottersten, “Optimal and suboptimal transmit beamforming,” in Handbook of Antennas in WirelessCommunications, CRC Press, 2001.
Javier Zazo Nonconvex QPs and games 5 / 28
Semidefinite programsReminder
I Linear program (LP):
minx
cTx
s.t. Gx ≤ 0
I Semidefinite program (SDP):
minx
cTx
s.t. F0 + x1F1 + . . .+ xNFN � 0
I Reduces to an LP if Fi are diagonal.I Can be optimally solved using specialized solvers.
Javier Zazo Nonconvex QPs and games 6 / 28
Semidefinite Relaxation of QPsI Given a QP:
minx∈Rp
xTA0x + 2bT0 x + c0
s.t. xTAix + 2bTi x + ci ≤ 0 ∀i = 1, . . . ,m,(1)
I Define X = xxT and transform xTAix = tr(AiX).
I Obtain non-convex QP:
minX∈Rp×p,x∈Rp
tr(A0X) + 2bT0 x + c0
s.t. tr(AiX) + 2bTi x + ci ≤ 0 ∀i = 1, . . . , N
X = xxT .
I Relax the rank constraint X � xTx � obtain an SDP:
minX∈Rp×p,x∈Rp
tr(A0X) + 2bT0 x + c0
s.t. tr(AiX) + 2bTi x + ci ≤ 0 ∀i = 1, . . . ,m
X � xTx.
(2)
I Strong duality: (1) and (2) attain the same solution.
Javier Zazo Nonconvex QPs and games 7 / 28
Trust region methods
I Problem: minimization of unconstrained problems
minx
f(x)
I Non-convex surrogate:
mind
dTBkd+ 2∇f(xk)T d
s.t. ‖d‖ ≤ ∆k,
where Bk is the Hessian of f(xk).
I QP with SINGLE quadratic constraint � presents STRONG duality.
minx
xTA0x + 2bT0 x + c0
s.t. g1(x) = xTA1x + 2bT1 x + c1 ≤ 0
Javier Zazo Nonconvex QPs and games 8 / 28
QP with a single equality constraint
minx
xTA0x + 2bT0 x + c0
s.t. g1(x) = xTA1x + 2bT1 x + c1 = 0
Application:
I Localization problems.
I Principal component analysis (PCA)
maxx
xTA0x
s.t. ‖x‖2 = 1.
Javier Zazo Nonconvex QPs and games 9 / 28
Outline
1 Nonconvex QPs with separable constraintsIntroduction to QPsRoadmap to establish strong duality in QPsRobust least squares with multiple constraints
2 Squared ranged localization problem
3 Algorithmic framework of QPs
4 Other works
5 Concluding remarks
QP with separable constraints
I QP with separable constraints:
minx
xTA0x + 2bT0 x + c0
s.t. xTAix + 2bTi x + ci E 0 ∀i = 1, . . . , N, N ≤ p,
where all constraints are separable and x = [x1, . . . , xN ], E ∈ {≤,= }.I Roadmap to establish strong duality:
S-propertyStrong alternatives
of SDPs
Strong alternativesof diagonalized SDP
QP w/ separableconstraints
TransformationQP ↔ SDP
Existence ofrank 1 solution
Javier Zazo Nonconvex QPs and games 10 / 28
Review of dual methodsI Consider a general optimization problem:
minx
f(x)
s.t. g(x) ≤ 0.
I Lagrangian: L(x,λ) = f(x) + λTg(x).
I Dual function:
q(λ) = minxL(x,λ)
I Dual problem:
maxλ≥0
q(λ)
Solve:
Update: λk+1 = [λk + αkg(xk)]+
arg minx L(x,λ)
Ou
ter
loo
p
Inn
erlo
op
I Karush-Kuhn-Tucker conditions (necessary):
∇xf(x) +∇xλTg(x) = 0
λTg(x) = 0
g(x) ≤ 0, λ ≥ 0
Javier Zazo Nonconvex QPs and games 11 / 28
Constraint Qualifications (CQs)
I CQs correspond to topological features on the feasible set.
I If satisfied, they guarantee the existence of dual variables for the KKT conditions.
I If not satisfied, dual variables do not exist that fulfill KKT conditions.
I Examples:
I Slater’s condition: gi(x) convex satisfies CQs if ∃x | gi(x) < 0.I Linear independence constraint qualification (LICQ):
gradients of the inequality constraints are linearly independent.I S-property :
I Defined as a system of equivalences.I Much more strict that Slater’s or LICQ � guarantees zero gap with dual problem.
Javier Zazo Nonconvex QPs and games 12 / 28
S-property & roadmap (recap)
Definition (S-property.)
A QP satisfies the S-property if and only if the following statements are equivalent for every α:
I ∀x feasible ⇒ f(x) ≥ αI ∃λ ∈ Γ |L(x,λ) ≥ α for all x ∈ Rp
I Roadmap to establish strong duality (recap)
S-propertyStrong alternatives
of SDPs
Strong alternativesof diagonalized SDP
QP w/ separableconstraints
TransformationQP ↔ SDP
Existence ofrank 1 solution
Javier Zazo Nonconvex QPs and games 13 / 28
Results on strong duality
A set of matrices {A1, A2, . . . , AN} is said to be simultaneously diagonalizable via congruence, if thereexists a nonsingular matrix P such that PTAiP is diagonal for every matrix Ai.
I Introduction of variables (from the QP):
Ai =
[Ai bibTi ci
], P ∈ Sp+1.
I Fi = PTAiP for all i ∈ { 1, . . . , N } become diagonal.
I F0 is not diagonal necessarily, only the constraints.
Theorem (Zazo et. al)
Given a QP with separable constraints, suppose Slater’s assumption is satisfied and that bi ∈ range[Ai]for every i ∈ N . Furthermore, assume there exists a diagonal matrix D whose elements are ±1 suchthat DF0D is a Z–matrix. Then, the S-property holds.
Javier Zazo Nonconvex QPs and games 14 / 28
Outline
1 Nonconvex QPs with separable constraintsIntroduction to QPsRoadmap to establish strong duality in QPsRobust least squares with multiple constraints
2 Squared ranged localization problem
3 Algorithmic framework of QPs
4 Other works
5 Concluding remarks
Robust least squares IApplication example
I Least squares problem:
minx
‖Ax− b‖2
I Robust least squares (RLS):
minx∈Rp
max(∆A,∆b)
‖(A+ ∆A)x− (b+ ∆b)‖2
s.t. ‖(∆A,∆b)‖2F ≤ ρ,
I Our proposal of RLS:
minx∈Rp
max(∆A,∆b)
‖(A+ ∆A)x− (b+ ∆b)‖2
s.t. ‖(∆A):i‖2 ≤ ρi ∀i ∈ { 1, . . . , p } ,‖∆b‖2 ≤ ρp+1
Javier Zazo Nonconvex QPs and games 15 / 28
How to solve a min-max problem
minx
maxy∈Y
φ(x, y)
I Proposed method:
1. Use (sub)gradient descent on the minimization variable.2. At each step, solve the maximization problem globally.
I We need to compute gradients of a maximization mapping. Define f(x) = maxy∈Y φ(x, y).
∇xf(x) = φ(x, y∗)
where y∗ = arg maxy∈Y f(x, y) (Danskin’s theorem).
I The maximization mapping is non-convex in the RLS problem.
I Can we solve it optimally? � We can try semidefinite relaxation.
Javier Zazo Nonconvex QPs and games 16 / 28
RLS: Strong duality result
Theorem (Zazo et. al)
Strong duality between primal problem and its dual holds for any H ∈ RN×p+1 andx = (xT ,−1)T ∈ Rp+1.
Sketch of proof:
1. Reformulate the objective function into standard form:
minx
xTA0x + 2bT0 x + c0
s.t. xTAix + 2bTi x + ci ≤ 0 ∀i = 1, . . . , N, N ≤ p
2. Show A0 is a completely positive matrix.
3. Determine matrix P , and compute F0 = PTA0P .
4. Verify there exists diagonal D such that DF0D is a completely positive matrix.
5. This satisfies requirements of previous theorem.
Javier Zazo Nonconvex QPs and games 17 / 28
Outline
1 Nonconvex QPs with separable constraints
2 Squared ranged localization problem
3 Algorithmic framework of QPs
4 Other works
5 Concluding remarks
Sensor network localization problem
I Problem formulation:
minx∈RNp
∑i∈Nu
∑j∈Na
i
(‖xi − sj‖2 − d2ij)
2 +∑
j∈Nui
(‖xi − xj‖2 − d2ij)
2.
I Network example:
−50 −40 −30 −20 −10 0 10 20 30 40 50
−20
−10
0
10
20 Anchor nodes
Unknown nodes
Estimates
Javier Zazo Nonconvex QPs and games 18 / 28
SimulationsN = 17 nodes with u. position. Noiseless and σ = 1. LOW CONNECTIVITY
0 50 100 150 20010−4
10−1
102
Iterations number
RM
SE
FLEXA poly 2
FLEXA poly 4
Costa et.al
Dual ascent
0 50 100 150 200100
101
102
Iterations number
RM
SE
FLEXA poly 2
FLEXA poly 4
Costa et.al
Dual ascent
Optimal solution
Javier Zazo Nonconvex QPs and games 19 / 28
SimulationsN = 11 nodes with u. position. Noiseless and σ = 1. HIGH CONNECTIVITY
0 50 100 150 20010−8
10−3
102
Iterations number
RM
SE
FLEXA poly 2
FLEXA poly 4
Costa et.al
Dual ascent
0 50 100 150 200
100
101
Iterations number
RM
SE
FLEXA poly 2
FLEXA poly 4
Costa et.al
Dual ascent
Optimal solution
Javier Zazo Nonconvex QPs and games 20 / 28
Outline
1 Nonconvex QPs with separable constraints
2 Squared ranged localization problem
3 Algorithmic framework of QPs
4 Other works
5 Concluding remarks
SDP Complexity
I SDP methods scale badly with the size of the problem � worst case complexity O(p5)!
I Descent techniques on primal problem may converge to local optima!
I We explore parallel techniques with optimality guarantees.
1. Projected gradient ascent method.2. Majorization-minimization methods.
Javier Zazo Nonconvex QPs and games 21 / 28
Projected dual (sub)gradient methodI Necessary condition for optimality:
A0 +∑i
λiAi � 0
I We defineW = {λ ∈ Γ | A0 +
∑iλiAi � 0 } .
I The dual problem is:
maxλ∈W
λTg(x(λ))
I Algorithm:
Solve:
Update: λk+1 = ΠW [λk + αkg(xk)]
arg minx L(x,λ)
Ou
ter
loo
p
Inn
erlo
op
Javier Zazo Nonconvex QPs and games 22 / 28
FLEXA decomposition IParallel updates
I FLEXA is a decomposition framework to solve non-convex problems in parallel.
I FLEXA solves problems of the following type:
minx
N∑i=1
fi(x)
s.t. g(x) ≤ 0
I The framework uses strongly convex surrogate functions:
G. Scutari, F. Facchinei, P. Song, D. P. Palomar, and J.-S. Pang, “Decomposition by partial linearization: Parallel optimization of multi-agentsystems,” IEEE TSP, vol. 62, no. 3, pp. 641–656, Feb. 2014
Javier Zazo Nonconvex QPs and games 23 / 28
Distributed proposal using an extra constraintI Alternative complementarity slackness problem:
A0 +∑
i∈NλiAi � 0, Z � 0, Z ⊥ A0 +∑
i∈NλiAi
I Dual problem:
minZ
tr[(A0 +
∑i∈N
λiAi
)Z]
s.t. Z � 0.
I Algorithm:
Solve:
Update: Zk+1 = [Zk + αk(A0 +∑
iλk+1i Ai)]+
arg minx,λ L(x,λ, Zk)
Ou
ter
loo
p
Inn
erlo
op
Javier Zazo Nonconvex QPs and games 24 / 28
Outline
1 Nonconvex QPs with separable constraints
2 Squared ranged localization problem
3 Algorithmic framework of QPs
4 Other works
5 Concluding remarks
Dynamic games
Agent 1 Agent Agent
Environment
I Formulated as Nash equilibrium (NE) problem.
I Two possible NE solution concepts:
1. Open loop NE.2. Closed loop NE.
Javier Zazo Nonconvex QPs and games 25 / 28
Demand-side management in smart grids
I Minimize energy costs.
I Acknowledge uncertainty in prize planning.
I Min-max game formulation � distributed framework.
Javier Zazo Nonconvex QPs and games 26 / 28
Outline
1 Nonconvex QPs with separable constraints
2 Squared ranged localization problem
3 Algorithmic framework of QPs
4 Other works
5 Concluding remarks
Conclusions
I Quadratic programs with separable constraints.
1. We study the conditions for QPs with separable constraints to exhibit the S-property.2. Novel algorithmic framework � New research lines.
I Squared ranged localization problems: TL and SNL problems
1. Primal methods (ADMM, NEXT, Diffusion)2. Dual methods (centralized and distributed)
I Algorithmic framework
1. Projected subgradient method � efficient but slow.2. Decomposition technique based on FLEXA � efficient and fast.
I Other works involving dynamic games and smart grid problems.
Javier Zazo Nonconvex QPs and games 27 / 28
Any questions??
Thank you!
Javier Zazo Nonconvex QPs and games 28 / 28
Top Related