Post on 18-Mar-2018
A Matlab Implementation of theMinpack-2 Test Problem Collection
Dr Shaun ForthCentre for Simulation and Analytics
S.A.Forth@cranfield.ac.uk
EuroAD Workshop, Paderborn Germany, 30 November 2015
Outline
1 Introduction
2 Matlab Implementation of Minpack-2 Problems
3 Examples
4 Testing
5 Results
6 Conclusions
7 Advert - AD20162/ 44
Introduction
1 IntroductionAD Packages in MatlabThe Minpack-2 Test Problem Collection
3/ 44
Introduction
Grad [Rich and Hill, 1992] - forward mode AD on a Matlab string viaTurbo-C
ADMAT [Verma, 1999] - forward and reverse (via tape) mode ADusing OO features of Matlab (+ second order derivatives and sparsitydetection)
ADiMat [Bischof et al., 2003] - source transformed forward/reversemode
MAD [Forth, 2006] - optimised derivatives storage class derivvec togive improved overloaded forward mode performance over ADMAT
MSAD [Kharche and Forth, 2006] - source-transformation byspecialising and inlining MAD’s derivative objects.
ADiGator [Patterson et al., 2013, Weinstein and Rao, 2015] -source-transformed, sparsity-exploiting forward mode AD (vertexelimination with forward ordering [Griewank and Reese, 1991]).
Need for testcases.4/ 44
The Minpack-2 Test ProblemCollection [Averick et al., 1991]
Describes 24 optimisation problems, including Fortran 77 sourcecode, of three problem types:
unconstrained minimisation - objective function, gradient,Hessian
least squares minimisation - residual, Jacobiansystems of nonlinear equations - residual, Jacobian
For large-scale problems source code for Hessian/Jacobian sparsitypattern and Hessian/Jacobian-vector products also provided.
Appears 82 times in Scopus
Widely used for AD tool validation, eg, Bischof et al. [1996], Waltherand Griewank [2004], Naumann and Utke [2005], Giering andKaminski [2006], Shin and Hovland [2007], Forth et al. [2004].
5/ 44
Matlab Implementation of Minpack-2Problems
2 Matlab Implementation of Minpack-2 ProblemsConverting Minpack-2 to MatlabRe-coding Minpack-2 in Matlab
6/ 44
Converting Minpack-2 to Matlab
Lenton [2005] hand-converted all the Minpack problems to MatlabI Changes of syntax, array constructorsI Arrays must have lower index 1 in Matlab (also affects loop indices)
Validated byI Fortran program creates random vectors x, calls Fortran Minpack,
writes f (x), ∇f , . . . to formatted text file.I Text file read into Matlab and results compared to those from Matlab
version of Minpack callsI Results agree to within i/o and floating point round-off
7/ 44
Re-coding Minpack-2 in Matlab
Lenton’s conversion satisfactory for small-scale, fixed n problems.
For large scale problems Lenton’s Fortran-based coding uses loopsand subscripting operations.
Large number of overloaded function calls in overloaded AD.
Re-coded Minpack-2 problems using array operations to give avectorised version.
Uniform interface to all functions with problem specific parameterssupplied in a structure.
8/ 44
Re-coding Minpack-2 in Matlab (ctd.)
Separate functions for:I Setting problem parameters - constants, standard start vector, etcI Function evaluation only + vectorised version where appropriate.I Function + Gradient/JacobianI Gradient/Jacobian onlyI Jacobian/Hessian-vector productI Sparsity pattern.
Use Matlab’s Code Analyzer to eliminate: dead code, unusedvariables.
Hand-coded adjoint for gradients.
9/ 44
Examples
3 ExamplesHuman Heart Dipole Problem (HHD)Minimal Surface Area (MSA) Problem
10/ 44
Human Heart Dipole Problem (HHD)
Calculating dipole moment of human heart, find x ∈ IR8 such thatF(x) = 0 with,
F(x)=
x1+x2−σmx
x3+x4−σmy
x5x1+x6x2−x7x3−x8x4−σA
x7x1+x8x2+x5x3+x6x4−σB
x1(x25−x2
7 )−2x3x5x7+x2(x26−x2
8 )−2x4u6u8−σC
x3(x25−x2
7 )+2x1x5x7+x4(x26−x2
8 )+2x2u6u8−σD
x1(x5x25−3v2
7 )+x3x7(x27−3v2
5 ) . . .. . .+x2x6(x2
6−3x28 )+x4x8(x2
8−3x26 )−σE
x3(x5x25−3v2
7 )−x1x7(x27−3v2
5 )+x4x6(x26−3x2
8 ) . . .. . .−x2x8(x2
8−3x26 )−σF ]
for given constant σmx , σmy , σA,. . . , σF .
11/ 44
HHD - Fortran Coding (much abbreviated)
subroutine dhhdfj(n,x,fvec,fjac,ldfjac,task,prob)
c set constants
if (prob .eq. 'DHHD1') then
summx = 0.485d0
:
else if (prob .eq. 'DHHD2') then
c standard start point
if (task .eq. 'XS') then
if (prob .eq. 'DHHD1') then
x(1) = 0.299d0
c intermediate variables
a = x(1)
c function
if (task .eq. 'F' .or. task .eq. 'FJ') then
fvec(1) = a + b - summx
c Jacobian
if (task .eq. 'J' .or. task .eq. 'FJ') then
fjac(1,1) = one
12/ 44
HHD - Lenton’s Matlab Conversion
function varargout=dhhdfj(x,task,prob)
if (prob == 'DHHD1')
summx = 0.485d0; ...
elseif (prob == 'DHHD2')
% Compute the standard starting point if task = 'XS'.
switch task
case 'XS'
if (prob == 'DHHD1')
x = [0.299d0; 0.186d0; -0.0273d0; 0.0254d0; ...
varargout{1}=x;
% function and/or Jacobian value
case {'F','J','FJ'}
if (task == 'F' | task == 'FJ')
fvec=[a + b - summx;... % array constructor
varargout{1}=fvec; % task dependant returned values
if (task == 'J' | task == 'FJ')
fjac= [ 1 1 0 0 0 0 0 0; ... % array constructor
varargout{nout}=fjac; % task dependant returned values
13/ 44
HHD - Re-Coding the Problem Definition
Structure Prob used to store all problem data
function Prob=MinpackHHD_Prob(vers)
Prob.user.vers = vers;
% set problem version dependent constants and x0
switch vers
case 1
Prob.user.summx = 0.485d0;
Prob.user.summy = -0.0019d0;
:
Prob.x_0 = [0.299d0; 0.186d0; -0.0273d0; 0.0254d0; ...
case 2
:
end
14/ 44
HHD - Re-Coding the Function Definition
For AD this is the code we would expect the user to supply.
function F=MinpackHHD_F(x,Prob)
% unpack constants
summx=Prob.user.summx ;
summy=Prob.user.summy ;
:
% intermediate variables
a = x(1);
b = x(2);
:
% Evaluate the function
F=[a + b - summx;
c + d - summy;
:
This will be the source code for our AD tests.
15/ 44
HHD - Recoding the Function and Jacobian
For many optimisation packages this is the ideal function definition.Jacobian only computed if required.
function [F,J]=MinpackHHD_FJ(x,Prob)
% unpack constants
summx=Prob.user.summx ;
summy=Prob.user.summy ;
:
% intermediate variables
a = x(1);
b = x(2);
F=[a + b - summx;
c + d - summy;
:
% Jacobian if required
if nargout==2
J= [ 1 1 0 0 0 0 0 0;
0 0 1 1 0 0 0 0;
:
16/ 44
HHD - Re-Coding the Jacobian only
Some packages require Jacobian computed separately from function
Remove unused variables
function J=MinpackHHD_J(x,Prob)
% constants - not required
% intermediate variables
a = x(1);
b = x(2);
:
% Jacobian
J= [ 1 1 0 0 0 0 0 0;
0 0 1 1 0 0 0 0;
:
17/ 44
Minimal Surface Area (MSA) Problem
Supply height on x and yboundaries of[−1
2 ,12 ]× [−1
2 ,12 ]
Determine surfaceu(x, y) of minimalsurface area with givenboundaries.
Known analytic solutiondue to Enneper.
18/ 44
Minimal Surface Area (MSA) Problem
Supply height on x and yboundaries of[−1
2 ,12 ]× [−1
2 ,12 ]
Determine surfaceu(x, y) of minimalsurface area with givenboundaries.
Known analytic solutiondue to Enneper.
18/ 44
Minimal Surface Area (MSA) Problem
Supply height on x and yboundaries of[−1
2 ,12 ]× [−1
2 ,12 ]
Determine surfaceu(x, y) of minimalsurface area with givenboundaries.
Known analytic solutiondue to Enneper.
18/ 44
MSA - Finite Difference Formulation
x
y
1 2 i i+1 ni +1 ni +21
2
j
j+1
nj +1
nj +2
Known boundary values
Unknown values
T Li,j
T Ui,j
Piecewise Linear function onT L,U
i,j
Min f(u) =∑ni+1i=1
∑nj+1j=1
(f Li,j + f U
i,j
)
19/ 44
MSA - Finite Difference Formulation
x
y
1 2 i i+1 ni +1 ni +21
2
j
j+1
nj +1
nj +2 Known boundary values
Unknown values
T Li,j
T Ui,j
Piecewise Linear function onT L,U
i,j
Min f(u) =∑ni+1i=1
∑nj+1j=1
(f Li,j + f U
i,j
)
19/ 44
MSA - Finite Difference Formulation
x
y
1 2 i i+1 ni +1 ni +21
2
j
j+1
nj +1
nj +2 Known boundary values
Unknown values
T Li,j
T Ui,j
Piecewise Linear function onT L,U
i,j
Min f(u) =∑ni+1i=1
∑nj+1j=1
(f Li,j + f U
i,j
)
19/ 44
MSA - Finite Difference Formulation
x
y
1 2 i i+1 ni +1 ni +21
2
j
j+1
nj +1
nj +2 Known boundary values
Unknown values
T Li,j
T Ui,j
Piecewise Linear function onT L,U
i,j
Min f(u) =∑ni+1i=1
∑nj+1j=1
(f Li,j + f U
i,j
)
19/ 44
MSA - Finite Difference Formulation
x
y
1 2 i i+1 ni +1 ni +21
2
j
j+1
nj +1
nj +2 Known boundary values
Unknown values
T Li,j
T Ui,j
Piecewise Linear function onT L,U
i,j
Min f(u) =∑ni+1i=1
∑nj+1j=1
(f Li,j + f U
i,j
)
19/ 44
MSA - Finite Difference Formulation (ctd.)
Minimise,
f (u) =ni+1∑i=1
nj+1∑j=1
(f Li,j + f U
i,j
),
with f L,Ui,j the surface area on the lower/upper triangles:
f Li,j =
hxhy
2
{1 +
(ui+1,j − ui,j
hx
)2
+
(ui,j+1 − ui,j
hy
)2} 1
2
f Ui,j =
hxhy
2
{1 +
(ui+1,j+1 − ui,j+1
hx
)2
+
(ui+1,j+1 − ui+1,j
hy
)2} 1
2
and we only consider ui,j with i = 2, . . . , ni + 1, j = 2, . . . , nj + 1.
20/ 44
MSA - Fortran Coding (much abbreviated)
subroutine dmsafg(nx,ny,x,f,fgrad,task,bottom,top,left,right)
:
c function and gradient over the lower triangular elements.
do 50 j = 0, ny
do 40 i = 0, nx
k = nx*(j-1) + i ! 1-D indexing
if (i .ge. 1 .and. j .ge. 1) then
v = x(k) ! first vertex in triangle
else
if (j .eq. 0) v = bottom(i+1)
:
if (i .lt. nx .and. j .gt. 0) then
vr = x(k+1) ! right vertex
:
dvdx = (vr-v)/hx
dvdy = (vt-v)/hy
fl = sqrt(one+dvdx**2+dvdy**2)
if (feval) f = f + fl
if (geval) then21/ 44
MSA - Lenton’s Matlab Conversion
function varargout=dmsafg(nx,ny,x,task,bottom,top,left,right)
:
switch task
case {'F','G','FG'}
for j = 0:ny
for i = 0:nx
k = nx*(j-1) + i; % 1-D indexing
if i >= 1 && j >= 1
v = x(k); % first vertex in triangle
else if j == 0
v = bottom(i+1);
:
if i < nx && j > 0
vr = x(k+1); % right vertex
:
dvdx = (vr-v)/hx;
dvdy = (vt-v)/hy;
fl = sqrt(1+dvdx^2+dvdy^2);
if geval22/ 44
MSA - Re-Coding the Problem DefinitionStructure Prob used to store all problem data
function [Prob,nuse]=MinpackMSA_Prob(varargin)
% [Prob,nuse]=MinpackMSA_Prob(nx,ny,bottom,top,left,right)
% check/set boundary conditions
if isempty(bottom)
bottom=Enneper('bottom',nx,ny);
elseif ~(isvector(bottom)&&length(bottom)==nx+2)
error(['MinPackMSA_Prob: bottom must be a length nx+2 = ',...
end
:
% Compute the standard starting point
x_0 = reshape((top(2:nx+1)*alpha + bottom(2:nx+1)*(1-alpha) +...
Prob.x_0=x_0;
Prob.user.nx=nx;
:
nuse=nx*ny;
function bcvec=Enneper(bc,nx,ny)
:23/ 44
MSA - Re-Coding the Function Definition
Regularise the interface to use Prob structure
function f = MinpackMSA_F(x,Prob)
:
bottom=Prob.user.bottom;
top =Prob.user.top;
left =Prob.user.left;
right =Prob.user.right;
otherwise similar to Lenton coding with loops and branching.
24/ 44
MSA - Vectorising the Function Definition
function f = MinpackMSA_Fvec(x,Prob)
:
bottom=Prob.user.bottom;
:
% transfer interior values x to entire grid v
v = zeros(nx+2,ny+2,'like',x); % 'like' ensures v has class of x
v(2:nx+1,2:ny+1) = reshape(x,nx,ny);
% apply boundary conditions
v(:,1) = bottom;
:
% computer dvdx and dvdy on each edge of the grid
dvdx = (v(2:nx+2,:)-v(1:nx+1,:))/hx;
dvdy = (v(:,2:ny+2)-v(:,1:ny+1))/hy;
% quadratic term over lower and upper elements
fL=sqrt(1+dvdx(1:nx+1,1:ny+1).^2+dvdy(1:nx+1,1:ny+1).^2);
fU=sqrt(1+dvdx(1:nx+1,2:ny+2).^2+dvdy(2:nx+2,1:ny+1).^2);
f = area*(sum(sum(fL+fU)));
No loops or branches!25/ 44
MSA - Vectorised Gradient
From vectorised function - easy (!) to write vectorised gradient
function fgrad = MinpackMSA_Gvec(x,Prob)
% coding as for MinpackMSA_Fvec
:
% quadratic term over lower and upper elements
fL=sqrt(1+dvdx(1:nx+1,1:ny+1).^2+dvdy(1:nx+1,1:ny+1).^2);
fU=sqrt(1+dvdx(1:nx+1,2:ny+2).^2+dvdy(2:nx+2,1:ny+1).^2);
% gradient just use interior points
i=2:nx+1;
j=2:ny+1;
fgrad=area*(...
(1/hx)*(1./fL(i-1,j)+1./fU(i-1,j-1)).*dvdx(i-1,j)...
-(1/hx)*(1./fL(i,j)+1./fU(i,j-1)).*dvdx(i,j)...
+(1/hy)*(1./fL(i,j-1)+1./fU(i-1,j-1)).*dvdy(i,j-1)...
-(1/hy)*(1./fL(i,j)+1./fU(i-1,j)).*dvdy(i,j));
fgrad=fgrad(:);
26/ 44
Testing
4 TestingVerification versus the Fortran ImplementationUnit Testing
27/ 44
Verification versus the Fortran Imple-mentation
Wish to validate Matlab coding against original Fortran coding.
Only Intel Visual Fortran can be used to compile Fortran Mex filescompatible with Matlab [Mathworks, 2015a].
Mex-file interface difficult to write.
Lenton [2005] usedI Formatted text file used to transfer data (eg, x) from Matlab to a Fortran
programI Fortran program evaluated F(x), JF(x), f (x), ∇f (x) etc.I Formatted text file used to transfer data back to Matlab.I Technique subject to truncation errors associated with I/O conversion.
Present approach uses (unformatted) binary files to eliminate I/Oconversion errors.
28/ 44
Unit Testing
Matlab’s unit test framework [Mathworks, 2015b] used to run tests onrandomised inputs.
Almost complete for Nonlinear Equations.
29/ 44
Unit Testing For Nonlinear EquationsCommon Features
Problem n version xs xl/xu F JF F + JFCPF 11 - X X X X XCPR 5 - X X X X XFDC 16,100 - X n/a X X XFIC 16,96 - X n/a X X XHHD 8 1-5 X X X X XIER 18,93 - X n/a X X XSFD 14,98 - X n/a X X XSFI 16,100 - X n/a X X X
30/ 44
Unit Testing For Nonlinear EquationsLargescale Features
Problem n S(JF) JF · vFDC 16,100 X XFIC 16,96 X XIER 18,93 X XSFD 14,98 X XSFI 16,100 X X
31/ 44
Unit Testing For Nonlinear EquationsVectorised Features
Problem n Fv JFv Fv + JFv Sv(JF) JF · vv
FDC 16,100 XFIC 16,96 XIER 18,93SFD 14,98SFI 16,100
32/ 44
Results
5 ResultsHHD ProblemMSA Problem
33/ 44
HHD Problem - Run time Ratios
Problem sizewhat how 8F Minpack 1F+G Minpack 0.5F+G fmad-sparse 200
Run time ratio =CPU(What by How)
CPU(F)
34/ 44
MSA Problem - Run time Ratios
Problem sizewhat how 100 400 1600F Minpack 1 1 1F+G Minpack 1.0000 2.2222 1.8235F+G fmad-sparse 5250 8889 19882Fvec vectorised 0.5000 0.4444 0.3529Fvec+G fmad-sparse 190.0000 120.0000 104.0000
35/ 44
Conclusions
6 Conclusions
36/ 44
Conclusions
We have a Matlab implementation of all problems in the Minpack-2Test Problem Collection.
Lenton’s implementation has been validated.
Moving to re-coded version with uniform interface and animplementation of the function alone to allow testing of AD for
I First Derivatives for Jacobians and gradientsI Sparsity Patterns for Jacobians and HessiansI Jacobian-vector and Hessian-vector productsI Impact of code vectorisation.
Code vectorisation will have a major impact on efficiency ofoverloaded and source transformed AD codes - includingcompile-time costs for source transformation.
37/ 44
Advert - AD2016
7 Advert - AD2016
38/ 44
Advert - AD20167th International Conference onAlgorithmic Differentiation
Monday 12th - Thursday 15th September 2016
Christ Church Oxford, UK
http://www.autodiff.org/ad16/
Key dates:I 1 January 2016 - Second Announcement and opening of abstract
submissionI 30 March 2016 - Deadline for submission of 4 page extended abstractsI 31 May 2016 - Notification of Acceptance of presentations and postersI 30 December 2016 - submission of papers to Optimization Methods
and Software
Please start thinking about your submission!
39/ 44
References I
B. M. Averick, R. G. Carter, and J. J. More. The Minpack-2 Test ProblemCollection. Technical Memorandum TM 150, Argonne National Laborartory,Mathematics and Computer Science Division, 9700 South Cass Avenue,Argonne, Illinois 60439, USA, may 1991. URLftp://ftp.mcs.anl.gov/pub/MINPACK-2/tprobs.92/P153.ps.Z.
C. Bischof, P. Khademi, A. Bouaricha, and A. Carle. Efficient computation ofgradients and jacobians by dynamic exploitation of sparsity in automaticdifferentiation. Optimization Methods and Software, 7(1):1–39, 1996. ISSN1055-6788.
C. Bischof, B. Lang, and A. Vehreschild. Automatic Differentiation for MATLABPrograms. Proceedings in Applied Mathematics and Mechanics, 2(1):50–53,mar 2003. ISSN 1617-7061. doi: 10.1002/pamm.200310013. URL http:
//onlinelibrary.wiley.com/doi/10.1002/pamm.200310013/abstract.
S. Forth, M. Tadjouddine, J. Pryce, and J. Reid. Jacobian code generated bysource transformation and vertex elimination can be as efficient ashand-coding. ACM Transactions on Mathematical Software, 30(3):266–299,2004. ISSN 0098-3500. doi: 10.1145/1024074.1024076.
40/ 44
References II
S. A. Forth. An efficient overloaded implementation of forward mode automaticdifferentiation in MATLAB. ACM Trans. Math. Softw., 32(2):195–222, jun 2006.ISSN 0098-3500. doi: 10.1145/1141885.1141888. URLhttp://doi.acm.org/10.1145/1141885.1141888.
R. Giering and T. Kaminski. Automatic Sparsity Detection implemented as asource-to-source transformation, volume 3994 LNCS - IV of Lecture Notes inComputer Science (including subseries Lecture Notes in Artificial Intelligenceand Lecture Notes in Bioinformatics). 2006. ISBN 3540343857.
A. Griewank and S. Reese. On the Calculation of Jacobian Matrices by theMarkowitz Rule. In A. Griewank and G. F. Corliss, editors, AutomaticDifferentiation of Algorithms - Theory, Implementation, and Application, pages126–135. SIAM, Philadelphia, PA, 1991. ISBN 0–89871–284–X.
R. Kharche and S. Forth. Source transformation for MATLAB automaticdifferentiation. In V. Alexandrov, G. van Albada, P. Sloot, and J. Dongarra,editors, Computational Science âAS ICCS 2006, volume 4 of Lecture Notes inComputer Science, pages 558–565. 2006. ISBN 978-3-540-34385-1. URLhttp://link.springer.com/chapter/10.1007%2F11758549_77.
41/ 44
References III
K. Lenton. An Efficient, Validated Implementation of the MINPACK-2 TestProblem Collection in MATLAB. MSc Dissertation, Cranfield University, AppliedMathematics \& Operational Research Group, Engineering SystemsDepartment, RMCS Shrivenham, Swindon SN6 8LA, UK, 2005.
Mathworks. Supported and Compatible Compilers for R2015b, 2015a. URLhttp://uk.mathworks.com/support/compilers/R2015b/index.html.
Mathworks. Unit Testing Framework, 2015b. URL http:
//uk.mathworks.com/help/matlab/matlab-unit-test-framework.html.
U. Naumann and J. Utke. Optimality-preserving elimination of linearities inJacobian accumulation. Electronic Transactions on Numerical Analysis, 21:134–150, 2005. ISSN 1068-9613.
M. A. Patterson, M. Weinstein, and A. V. Rao. An efficient overloaded method forcomputing derivatives of mathematical functions in MATLAB. ACM Trans.Math. Softw., 39(3):17:1–17:36, may 2013. ISSN 0098-3500. doi:10.1145/2450153.2450155. URLhttp://doi.acm.org/10.1145/2450153.2450155.
42/ 44
References IV
L. Rich and D. Hill. Automatic differentiation in MATLAB. Applied NumericalMathematics, 9(1):33–43, 1992. ISSN 0168-9274. doi:10.1016/0168-9274(92)90065-L.
J. Shin and P. Hovland. Comparison of two activity analyses for automaticdifferentiation: Context-sensitive flow-insensitive vs. context-insensitiveflow-sensitive. Proceedings of the ACM Symposium on Applied Computing,pages 1323–1329, 2007. doi: 10.1145/1244002.1244287.
A. Verma. ADMAT: Automatic Differentiation in MATLAB Using Object OrientedMethods. In M. E. Henderson, C. R. Anderson, and S. L. Lyons, editors, ObjectOriented Methods for Interoperable Scientific and Engineering Computing:Proceedings of the 1998 SIAM Workshop, pages 174–183. SIAM,Philadelphia, 1999.
A. Walther and A. Griewank. ADOL-C: Computing higher-order derivatives andsparsity pattern for functions written in C/C++. 2004.
43/ 44
References V
M. J. Weinstein and A. V. Rao. ADiGator, a Toolbox for the AlgorithmicDifferentiation of Mathematical Functions in MATLAB Using SourceTransformation via Operator Overloading. ACM Transactions on MathematicalSoftware, Submitted, 2015. URL http://vdol.mae.ufl.edu/
SubmittedJournalPublications/adigator-CALGO.pdf.
44/ 44