Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department...
-
date post
21-Dec-2015 -
Category
Documents
-
view
218 -
download
0
Transcript of Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department...
![Page 1: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/1.jpg)
Quadratic Minimisation Problems in Statistics
Casper Albers, Frank Critchley & John Gower
Department of Statistics, The Open University
![Page 2: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/2.jpg)
Outline
• Introduction to problem (1)• Statistical examples of problem (1)• Geometrical insights: some easy, some hard• Concluding remarks
![Page 3: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/3.jpg)
The essential problem
• A and B are square matrices (of the same order p)• A is p.d. or p.s.d.• B can be anything• The constraint is consistent
(1)
![Page 4: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/4.jpg)
Equivalent forms
Eq. (1) can occur in many other shapes and forms, e.g.:
• min (x – t)′A(x – t) subject to (x-s)′B(x-s) + 2g′(x-s) = k
• minx ||Xx – y||2 subject to x′Bx + 2b′x = k
• min trace (X – T)′A(X – T)
subject to trace (X′BX + 2G′X) = k
• We present a unified solution to all such problems.
![Page 5: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/5.jpg)
General canonical form
• After simple affine transformations z = T-1 x + m and s = T-1 t + m where T is such that,
, (1) reduces to:
0
1',Γ
ΓBTT
0
IATT
'
k
zgzz
szz
'2'
:subject to
||||min 2
![Page 6: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/6.jpg)
Applications
Problem (1) arises, for example, in:• Canonical analysis• Normal linear models with quadratic constraints• The fitting of cubic splines to a cloud of points• Various forms of oblique Procrustes analysis• Procrustes analysis with missing values• Bayesian decision theory under quadratic loss• Minimum distance estimation• Hardy-Weinberg estimation• Updating ALSCAL algorithm• …
![Page 7: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/7.jpg)
Application: Hardy-Weinberg
• Genotypes AA, BB, AB in proportions p = (p1, p2, p3)
• Observed proportions q = (q1, q2, q3)
• HW equilibrium constraint p32 = 4 p1 p2
• Additional constraints: 1′ p = 1, p ≥ 0• GCF:
Note linear term
61
1362
2
222
211
subject to
min
zz
szszz
![Page 8: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/8.jpg)
Indefinite constrained regression
• Ten Berge (1983) considers for the ALSCAL algorithm:
• The GCF has eigenvalues:
(1 + √2, ½, 1 - √2)
![Page 9: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/9.jpg)
Ratios of quadratic forms (1)• Canonical analysis: min x′Wx / x′Bx.
• When W or B is of full rank, we have:
min x′Wx s.t. x′Bx = 1, of form (1) with
Lagrangian Wx = λBx.
• BUT: the ratio form requires only a weak constraint while if the Lagrangian is taken as fundamental, the constraint becomes strong (see Healy & Goldstein, 1976, for x′1 = 1).
• In canonical analysis, multiple solutions are standard but seem to have no place in our more general problem (1).
![Page 10: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/10.jpg)
Ratios of quadratic forms (2)
When both A and B are of deficient rank:
• In the canonical case, the ANOVA T = W + B implies that the null space of T is shared by B and W, and a simple modification of the usual two-sided eigenvalue solution suffices.
• However, for general matrices A, B things become much more complicated.
![Page 11: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/11.jpg)
Geometry helps understanding
The following slides illustrate the problem geometrically showing some of the complications that have to be covered by the algebra and algorithms.
![Page 12: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/12.jpg)
PD and indefinite case
B is positive definite B is indefinite
![Page 13: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/13.jpg)
Lower dimensional target space
![Page 14: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/14.jpg)
Lower dimensional target space
![Page 15: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/15.jpg)
Indefinite constraints
Full dimensionaltarget space
Lower dimensionaltarget space
![Page 16: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/16.jpg)
Parabola
![Page 17: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/17.jpg)
Projections onto target space
B not canonical B canonical
![Page 18: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/18.jpg)
Fundamental Canonical Form
• (1) boils down to minz ||z – s||2 subject to z′ Γ z = k
• This gives Lagrangian form: ||z – s||2 – λ(z′ Γ z – k)• With z = (I – λ Γ)-1 s, the constraint becomes
• In general, solutions found by solving this Lagrangian• Feasible region (FR):
– When B is indefinite: 1/γ1 ≤ λ ≤ 1/γp
– When B is p.(s.)d.: –∞ ≤ λ ≤ 1/γp
– f(λ) increases monotonically in the FR
• If s1 or sp are zero, adaptations are necessary
![Page 19: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/19.jpg)
Lagrangian forms
B indefinite B p.(s.)d.
![Page 20: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/20.jpg)
Lagrangian forms: phantom asymptotes
s2 = 0s1 = 0
root
![Page 21: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/21.jpg)
Movement from the origin
![Page 22: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/22.jpg)
Movement from the origin
![Page 23: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/23.jpg)
Movement from the origin
![Page 24: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/24.jpg)
Movement along the major axis
![Page 25: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/25.jpg)
Movement along the major axis
![Page 26: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/26.jpg)
Conclusions
• Equation (1) subsumes many statistical problems.• A unified methodology eliminates examination of many
special cases.• Geometry helps understanding; algebra helps detailed
analysis and provides essential underpinning for a general purpose algorithm.
• By identifying potential pathological situations, the algorithm can
• be made robust• provide warnings.
![Page 27: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/27.jpg)
Conclusions (informal)
• The unification is interesting and potentially useful.
• Its usefulness largely depends on the availability of a general purpose algorithm. Coming soon.
• Algorithms depend on detailed algebraic underpinning Done.
• Developing the algebra depends on understanding the geometry. Done
![Page 28: Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University.](https://reader035.fdocuments.net/reader035/viewer/2022062516/56649d5f5503460f94a3f69c/html5/thumbnails/28.jpg)
Some references
• C.J. Albers, F. Critchley, J.C. Gower, Quadratic Minimisation Problems in Statistics, 21st century
• M.W. Browne, On oblique Procrustes rotation, Psychometrika 32, 1967
• J.M.F. ten Berge, A generalization of Verhelst’s solution for a constrained regression problem in ALSCAL and related MDS algorithms, Psychometrika 48, 1983
• F. Critchley, On the minimisation of a positive definite quadratic form under quadratic constraints: analytical solution and statistical applications. Warwick Statistics Research Report, 1990
• M.J.R. Healy and H. Goldstein, An approach to the scaling of categorical attributes, Biometrika 63, 1976
• J. de Leeuw, Generalized eigenvalue problems with psd matrices, Psychometrika 47, 1982
• J.J. Moré, Generalizations of the trust region problem, Optimization methods and software, Vol. II, 1993
• J.C. Gower & G.B. Dijksterhuis, Procrustes Problems, Oxford University Press, 2004