A Matlab Implementation of theMinpack-2 Test Problem Collection

Dr Shaun ForthCentre for Simulation and Analytics

S.A.Forth@cranfield.ac.uk

EuroAD Workshop, Paderborn Germany, 30 November 2015

Outline

1 Introduction

2 Matlab Implementation of Minpack-2 Problems

3 Examples

4 Testing

5 Results

6 Conclusions

7 Advert - AD20162/ 44

Introduction

1 IntroductionAD Packages in MatlabThe Minpack-2 Test Problem Collection

3/ 44

Introduction

Grad [Rich and Hill, 1992] - forward mode AD on a Matlab string viaTurbo-C

ADMAT [Verma, 1999] - forward and reverse (via tape) mode ADusing OO features of Matlab (+ second order derivatives and sparsitydetection)

ADiMat [Bischof et al., 2003] - source transformed forward/reversemode

MAD [Forth, 2006] - optimised derivatives storage class derivvec togive improved overloaded forward mode performance over ADMAT

MSAD [Kharche and Forth, 2006] - source-transformation byspecialising and inlining MAD’s derivative objects.

ADiGator [Patterson et al., 2013, Weinstein and Rao, 2015] -source-transformed, sparsity-exploiting forward mode AD (vertexelimination with forward ordering [Griewank and Reese, 1991]).

Need for testcases.4/ 44

The Minpack-2 Test ProblemCollection [Averick et al., 1991]

Describes 24 optimisation problems, including Fortran 77 sourcecode, of three problem types:

unconstrained minimisation - objective function, gradient,Hessian

least squares minimisation - residual, Jacobiansystems of nonlinear equations - residual, Jacobian

For large-scale problems source code for Hessian/Jacobian sparsitypattern and Hessian/Jacobian-vector products also provided.

Appears 82 times in Scopus

Widely used for AD tool validation, eg, Bischof et al. [1996], Waltherand Griewank [2004], Naumann and Utke [2005], Giering andKaminski [2006], Shin and Hovland [2007], Forth et al. [2004].

5/ 44

Matlab Implementation of Minpack-2Problems

2 Matlab Implementation of Minpack-2 ProblemsConverting Minpack-2 to MatlabRe-coding Minpack-2 in Matlab

6/ 44

Converting Minpack-2 to Matlab

Lenton [2005] hand-converted all the Minpack problems to MatlabI Changes of syntax, array constructorsI Arrays must have lower index 1 in Matlab (also affects loop indices)

Validated byI Fortran program creates random vectors x, calls Fortran Minpack,

writes f (x), ∇f , . . . to formatted text file.I Text file read into Matlab and results compared to those from Matlab

version of Minpack callsI Results agree to within i/o and floating point round-off

7/ 44

Re-coding Minpack-2 in Matlab

Lenton’s conversion satisfactory for small-scale, fixed n problems.

For large scale problems Lenton’s Fortran-based coding uses loopsand subscripting operations.

Large number of overloaded function calls in overloaded AD.

Re-coded Minpack-2 problems using array operations to give avectorised version.

Uniform interface to all functions with problem specific parameterssupplied in a structure.

8/ 44

Re-coding Minpack-2 in Matlab (ctd.)

Separate functions for:I Setting problem parameters - constants, standard start vector, etcI Function evaluation only + vectorised version where appropriate.I Function + Gradient/JacobianI Gradient/Jacobian onlyI Jacobian/Hessian-vector productI Sparsity pattern.

Use Matlab’s Code Analyzer to eliminate: dead code, unusedvariables.

Hand-coded adjoint for gradients.

9/ 44

Examples

3 ExamplesHuman Heart Dipole Problem (HHD)Minimal Surface Area (MSA) Problem

10/ 44

Human Heart Dipole Problem (HHD)

Calculating dipole moment of human heart, find x ∈ IR8 such thatF(x) = 0 with,

F(x)=

x1+x2−σmx

x3+x4−σmy

x5x1+x6x2−x7x3−x8x4−σA

x7x1+x8x2+x5x3+x6x4−σB

x1(x25−x2

7 )−2x3x5x7+x2(x26−x2

8 )−2x4u6u8−σC

x3(x25−x2

7 )+2x1x5x7+x4(x26−x2

8 )+2x2u6u8−σD

x1(x5x25−3v2

7 )+x3x7(x27−3v2

5 ) . . .. . .+x2x6(x2

6−3x28 )+x4x8(x2

8−3x26 )−σE

x3(x5x25−3v2

7 )−x1x7(x27−3v2

5 )+x4x6(x26−3x2

8 ) . . .. . .−x2x8(x2

8−3x26 )−σF ]

for given constant σmx , σmy , σA,. . . , σF .

11/ 44

HHD - Fortran Coding (much abbreviated)

subroutine dhhdfj(n,x,fvec,fjac,ldfjac,task,prob)

c set constants

if (prob .eq. 'DHHD1') then

summx = 0.485d0

:

else if (prob .eq. 'DHHD2') then

c standard start point

if (task .eq. 'XS') then

if (prob .eq. 'DHHD1') then

x(1) = 0.299d0

c intermediate variables

a = x(1)

c function

if (task .eq. 'F' .or. task .eq. 'FJ') then

fvec(1) = a + b - summx

c Jacobian

if (task .eq. 'J' .or. task .eq. 'FJ') then

fjac(1,1) = one

12/ 44

HHD - Lenton’s Matlab Conversion

function varargout=dhhdfj(x,task,prob)

if (prob == 'DHHD1')

summx = 0.485d0; ...

elseif (prob == 'DHHD2')

% Compute the standard starting point if task = 'XS'.

switch task

case 'XS'

if (prob == 'DHHD1')

x = [0.299d0; 0.186d0; -0.0273d0; 0.0254d0; ...

varargout{1}=x;

% function and/or Jacobian value

case {'F','J','FJ'}

if (task == 'F' | task == 'FJ')

fvec=[a + b - summx;... % array constructor

varargout{1}=fvec; % task dependant returned values

if (task == 'J' | task == 'FJ')

fjac= [ 1 1 0 0 0 0 0 0; ... % array constructor

varargout{nout}=fjac; % task dependant returned values

13/ 44

HHD - Re-Coding the Problem Definition

Structure Prob used to store all problem data

function Prob=MinpackHHD_Prob(vers)

Prob.user.vers = vers;

% set problem version dependent constants and x0

switch vers

case 1

Prob.user.summx = 0.485d0;

Prob.user.summy = -0.0019d0;

:

Prob.x_0 = [0.299d0; 0.186d0; -0.0273d0; 0.0254d0; ...

case 2

:

end

14/ 44

HHD - Re-Coding the Function Definition

For AD this is the code we would expect the user to supply.

function F=MinpackHHD_F(x,Prob)

% unpack constants

summx=Prob.user.summx ;

summy=Prob.user.summy ;

:

% intermediate variables

a = x(1);

b = x(2);

:

% Evaluate the function

F=[a + b - summx;

c + d - summy;

:

This will be the source code for our AD tests.

15/ 44

HHD - Recoding the Function and Jacobian

For many optimisation packages this is the ideal function definition.Jacobian only computed if required.

function [F,J]=MinpackHHD_FJ(x,Prob)

% unpack constants

summx=Prob.user.summx ;

summy=Prob.user.summy ;

:

% intermediate variables

a = x(1);

b = x(2);

F=[a + b - summx;

c + d - summy;

:

% Jacobian if required

if nargout==2

J= [ 1 1 0 0 0 0 0 0;

0 0 1 1 0 0 0 0;

:

16/ 44

HHD - Re-Coding the Jacobian only

Some packages require Jacobian computed separately from function

Remove unused variables

function J=MinpackHHD_J(x,Prob)

% constants - not required

% intermediate variables

a = x(1);

b = x(2);

:

% Jacobian

J= [ 1 1 0 0 0 0 0 0;

0 0 1 1 0 0 0 0;

:

17/ 44

Minimal Surface Area (MSA) Problem

Supply height on x and yboundaries of[−1

2 ,12 ]× [−1

2 ,12 ]

Determine surfaceu(x, y) of minimalsurface area with givenboundaries.

Known analytic solutiondue to Enneper.

18/ 44

Minimal Surface Area (MSA) Problem

Supply height on x and yboundaries of[−1

2 ,12 ]× [−1

2 ,12 ]

Determine surfaceu(x, y) of minimalsurface area with givenboundaries.

Known analytic solutiondue to Enneper.

18/ 44

Minimal Surface Area (MSA) Problem

Supply height on x and yboundaries of[−1

2 ,12 ]× [−1

2 ,12 ]

Determine surfaceu(x, y) of minimalsurface area with givenboundaries.

Known analytic solutiondue to Enneper.

18/ 44

MSA - Finite Difference Formulation

x

y

1 2 i i+1 ni +1 ni +21

2

j

j+1

nj +1

nj +2

Known boundary values

Unknown values

T Li,j

T Ui,j

Piecewise Linear function onT L,U

i,j

Min f(u) =∑ni+1i=1

∑nj+1j=1

(f Li,j + f U

i,j

)

19/ 44

MSA - Finite Difference Formulation

x

y

1 2 i i+1 ni +1 ni +21

2

j

j+1

nj +1

nj +2 Known boundary values

Unknown values

T Li,j

T Ui,j

Piecewise Linear function onT L,U

i,j

Min f(u) =∑ni+1i=1

∑nj+1j=1

(f Li,j + f U

i,j

)

19/ 44

MSA - Finite Difference Formulation

x

y

1 2 i i+1 ni +1 ni +21

2

j

j+1

nj +1

nj +2 Known boundary values

Unknown values

T Li,j

T Ui,j

Piecewise Linear function onT L,U

i,j

Min f(u) =∑ni+1i=1

∑nj+1j=1

(f Li,j + f U

i,j

)

19/ 44

MSA - Finite Difference Formulation

x

y

1 2 i i+1 ni +1 ni +21

2

j

j+1

nj +1

nj +2 Known boundary values

Unknown values

T Li,j

T Ui,j

Piecewise Linear function onT L,U

i,j

Min f(u) =∑ni+1i=1

∑nj+1j=1

(f Li,j + f U

i,j

)

19/ 44

MSA - Finite Difference Formulation

x

y

1 2 i i+1 ni +1 ni +21

2

j

j+1

nj +1

nj +2 Known boundary values

Unknown values

T Li,j

T Ui,j

Piecewise Linear function onT L,U

i,j

Min f(u) =∑ni+1i=1

∑nj+1j=1

(f Li,j + f U

i,j

)

19/ 44

MSA - Finite Difference Formulation (ctd.)

Minimise,

f (u) =ni+1∑i=1

nj+1∑j=1

(f Li,j + f U

i,j

),

with f L,Ui,j the surface area on the lower/upper triangles:

f Li,j =

hxhy

2

{1 +

(ui+1,j − ui,j

hx

)2

+

(ui,j+1 − ui,j

hy

)2} 1

2

f Ui,j =

hxhy

2

{1 +

(ui+1,j+1 − ui,j+1

hx

)2

+

(ui+1,j+1 − ui+1,j

hy

)2} 1

2

and we only consider ui,j with i = 2, . . . , ni + 1, j = 2, . . . , nj + 1.

20/ 44

MSA - Fortran Coding (much abbreviated)

subroutine dmsafg(nx,ny,x,f,fgrad,task,bottom,top,left,right)

:

c function and gradient over the lower triangular elements.

do 50 j = 0, ny

do 40 i = 0, nx

k = nx*(j-1) + i ! 1-D indexing

if (i .ge. 1 .and. j .ge. 1) then

v = x(k) ! first vertex in triangle

else

if (j .eq. 0) v = bottom(i+1)

:

if (i .lt. nx .and. j .gt. 0) then

vr = x(k+1) ! right vertex

:

dvdx = (vr-v)/hx

dvdy = (vt-v)/hy

fl = sqrt(one+dvdx**2+dvdy**2)

if (feval) f = f + fl

if (geval) then21/ 44

MSA - Lenton’s Matlab Conversion

function varargout=dmsafg(nx,ny,x,task,bottom,top,left,right)

:

switch task

case {'F','G','FG'}

for j = 0:ny

for i = 0:nx

k = nx*(j-1) + i; % 1-D indexing

if i >= 1 && j >= 1

v = x(k); % first vertex in triangle

else if j == 0

v = bottom(i+1);

:

if i < nx && j > 0

vr = x(k+1); % right vertex

:

dvdx = (vr-v)/hx;

dvdy = (vt-v)/hy;

fl = sqrt(1+dvdx^2+dvdy^2);

if geval22/ 44

MSA - Re-Coding the Problem DefinitionStructure Prob used to store all problem data

function [Prob,nuse]=MinpackMSA_Prob(varargin)

% [Prob,nuse]=MinpackMSA_Prob(nx,ny,bottom,top,left,right)

% check/set boundary conditions

if isempty(bottom)

bottom=Enneper('bottom',nx,ny);

elseif ~(isvector(bottom)&&length(bottom)==nx+2)

error(['MinPackMSA_Prob: bottom must be a length nx+2 = ',...

end

:

% Compute the standard starting point

x_0 = reshape((top(2:nx+1)*alpha + bottom(2:nx+1)*(1-alpha) +...

Prob.x_0=x_0;

Prob.user.nx=nx;

:

nuse=nx*ny;

function bcvec=Enneper(bc,nx,ny)

:23/ 44

MSA - Re-Coding the Function Definition

Regularise the interface to use Prob structure

function f = MinpackMSA_F(x,Prob)

:

bottom=Prob.user.bottom;

top =Prob.user.top;

left =Prob.user.left;

right =Prob.user.right;

otherwise similar to Lenton coding with loops and branching.

24/ 44

MSA - Vectorising the Function Definition

function f = MinpackMSA_Fvec(x,Prob)

:

bottom=Prob.user.bottom;

:

% transfer interior values x to entire grid v

v = zeros(nx+2,ny+2,'like',x); % 'like' ensures v has class of x

v(2:nx+1,2:ny+1) = reshape(x,nx,ny);

% apply boundary conditions

v(:,1) = bottom;

:

% computer dvdx and dvdy on each edge of the grid

dvdx = (v(2:nx+2,:)-v(1:nx+1,:))/hx;

dvdy = (v(:,2:ny+2)-v(:,1:ny+1))/hy;

% quadratic term over lower and upper elements

fL=sqrt(1+dvdx(1:nx+1,1:ny+1).^2+dvdy(1:nx+1,1:ny+1).^2);

fU=sqrt(1+dvdx(1:nx+1,2:ny+2).^2+dvdy(2:nx+2,1:ny+1).^2);

f = area*(sum(sum(fL+fU)));

No loops or branches!25/ 44

MSA - Vectorised Gradient

From vectorised function - easy (!) to write vectorised gradient

function fgrad = MinpackMSA_Gvec(x,Prob)

% coding as for MinpackMSA_Fvec

:

% quadratic term over lower and upper elements

fL=sqrt(1+dvdx(1:nx+1,1:ny+1).^2+dvdy(1:nx+1,1:ny+1).^2);

fU=sqrt(1+dvdx(1:nx+1,2:ny+2).^2+dvdy(2:nx+2,1:ny+1).^2);

% gradient just use interior points

i=2:nx+1;

j=2:ny+1;

fgrad=area*(...

(1/hx)*(1./fL(i-1,j)+1./fU(i-1,j-1)).*dvdx(i-1,j)...

-(1/hx)*(1./fL(i,j)+1./fU(i,j-1)).*dvdx(i,j)...

+(1/hy)*(1./fL(i,j-1)+1./fU(i-1,j-1)).*dvdy(i,j-1)...

-(1/hy)*(1./fL(i,j)+1./fU(i-1,j)).*dvdy(i,j));

fgrad=fgrad(:);

26/ 44

Testing

4 TestingVerification versus the Fortran ImplementationUnit Testing

27/ 44

Verification versus the Fortran Imple-mentation

Wish to validate Matlab coding against original Fortran coding.

Only Intel Visual Fortran can be used to compile Fortran Mex filescompatible with Matlab [Mathworks, 2015a].

Mex-file interface difficult to write.

Lenton [2005] usedI Formatted text file used to transfer data (eg, x) from Matlab to a Fortran

programI Fortran program evaluated F(x), JF(x), f (x), ∇f (x) etc.I Formatted text file used to transfer data back to Matlab.I Technique subject to truncation errors associated with I/O conversion.

Present approach uses (unformatted) binary files to eliminate I/Oconversion errors.

28/ 44

Unit Testing

Matlab’s unit test framework [Mathworks, 2015b] used to run tests onrandomised inputs.

Almost complete for Nonlinear Equations.

29/ 44

Unit Testing For Nonlinear EquationsCommon Features

Problem n version xs xl/xu F JF F + JFCPF 11 - X X X X XCPR 5 - X X X X XFDC 16,100 - X n/a X X XFIC 16,96 - X n/a X X XHHD 8 1-5 X X X X XIER 18,93 - X n/a X X XSFD 14,98 - X n/a X X XSFI 16,100 - X n/a X X X

30/ 44

Unit Testing For Nonlinear EquationsLargescale Features

Problem n S(JF) JF · vFDC 16,100 X XFIC 16,96 X XIER 18,93 X XSFD 14,98 X XSFI 16,100 X X

31/ 44

Unit Testing For Nonlinear EquationsVectorised Features

Problem n Fv JFv Fv + JFv Sv(JF) JF · vv

FDC 16,100 XFIC 16,96 XIER 18,93SFD 14,98SFI 16,100

32/ 44

Results

5 ResultsHHD ProblemMSA Problem

33/ 44

HHD Problem - Run time Ratios

Problem sizewhat how 8F Minpack 1F+G Minpack 0.5F+G fmad-sparse 200

Run time ratio =CPU(What by How)

CPU(F)

34/ 44

MSA Problem - Run time Ratios

Problem sizewhat how 100 400 1600F Minpack 1 1 1F+G Minpack 1.0000 2.2222 1.8235F+G fmad-sparse 5250 8889 19882Fvec vectorised 0.5000 0.4444 0.3529Fvec+G fmad-sparse 190.0000 120.0000 104.0000

35/ 44

Conclusions

6 Conclusions

36/ 44

Conclusions

We have a Matlab implementation of all problems in the Minpack-2Test Problem Collection.

Lenton’s implementation has been validated.

Moving to re-coded version with uniform interface and animplementation of the function alone to allow testing of AD for

I First Derivatives for Jacobians and gradientsI Sparsity Patterns for Jacobians and HessiansI Jacobian-vector and Hessian-vector productsI Impact of code vectorisation.

Code vectorisation will have a major impact on efficiency ofoverloaded and source transformed AD codes - includingcompile-time costs for source transformation.

37/ 44

Advert - AD2016

7 Advert - AD2016

38/ 44

Advert - AD20167th International Conference onAlgorithmic Differentiation

Monday 12th - Thursday 15th September 2016

Christ Church Oxford, UK

http://www.autodiff.org/ad16/

Key dates:I 1 January 2016 - Second Announcement and opening of abstract

submissionI 30 March 2016 - Deadline for submission of 4 page extended abstractsI 31 May 2016 - Notification of Acceptance of presentations and postersI 30 December 2016 - submission of papers to Optimization Methods

and Software

Please start thinking about your submission!

39/ 44

References I

B. M. Averick, R. G. Carter, and J. J. More. The Minpack-2 Test ProblemCollection. Technical Memorandum TM 150, Argonne National Laborartory,Mathematics and Computer Science Division, 9700 South Cass Avenue,Argonne, Illinois 60439, USA, may 1991. URLftp://ftp.mcs.anl.gov/pub/MINPACK-2/tprobs.92/P153.ps.Z.

C. Bischof, P. Khademi, A. Bouaricha, and A. Carle. Efficient computation ofgradients and jacobians by dynamic exploitation of sparsity in automaticdifferentiation. Optimization Methods and Software, 7(1):1–39, 1996. ISSN1055-6788.

C. Bischof, B. Lang, and A. Vehreschild. Automatic Differentiation for MATLABPrograms. Proceedings in Applied Mathematics and Mechanics, 2(1):50–53,mar 2003. ISSN 1617-7061. doi: 10.1002/pamm.200310013. URL http:

//onlinelibrary.wiley.com/doi/10.1002/pamm.200310013/abstract.

S. Forth, M. Tadjouddine, J. Pryce, and J. Reid. Jacobian code generated bysource transformation and vertex elimination can be as efficient ashand-coding. ACM Transactions on Mathematical Software, 30(3):266–299,2004. ISSN 0098-3500. doi: 10.1145/1024074.1024076.

40/ 44

References II

S. A. Forth. An efficient overloaded implementation of forward mode automaticdifferentiation in MATLAB. ACM Trans. Math. Softw., 32(2):195–222, jun 2006.ISSN 0098-3500. doi: 10.1145/1141885.1141888. URLhttp://doi.acm.org/10.1145/1141885.1141888.

R. Giering and T. Kaminski. Automatic Sparsity Detection implemented as asource-to-source transformation, volume 3994 LNCS - IV of Lecture Notes inComputer Science (including subseries Lecture Notes in Artificial Intelligenceand Lecture Notes in Bioinformatics). 2006. ISBN 3540343857.

A. Griewank and S. Reese. On the Calculation of Jacobian Matrices by theMarkowitz Rule. In A. Griewank and G. F. Corliss, editors, AutomaticDifferentiation of Algorithms - Theory, Implementation, and Application, pages126–135. SIAM, Philadelphia, PA, 1991. ISBN 0–89871–284–X.

R. Kharche and S. Forth. Source transformation for MATLAB automaticdifferentiation. In V. Alexandrov, G. van Albada, P. Sloot, and J. Dongarra,editors, Computational Science âAS ICCS 2006, volume 4 of Lecture Notes inComputer Science, pages 558–565. 2006. ISBN 978-3-540-34385-1. URLhttp://link.springer.com/chapter/10.1007%2F11758549_77.

41/ 44

References III

K. Lenton. An Efficient, Validated Implementation of the MINPACK-2 TestProblem Collection in MATLAB. MSc Dissertation, Cranfield University, AppliedMathematics \& Operational Research Group, Engineering SystemsDepartment, RMCS Shrivenham, Swindon SN6 8LA, UK, 2005.

Mathworks. Supported and Compatible Compilers for R2015b, 2015a. URLhttp://uk.mathworks.com/support/compilers/R2015b/index.html.

Mathworks. Unit Testing Framework, 2015b. URL http:

//uk.mathworks.com/help/matlab/matlab-unit-test-framework.html.

U. Naumann and J. Utke. Optimality-preserving elimination of linearities inJacobian accumulation. Electronic Transactions on Numerical Analysis, 21:134–150, 2005. ISSN 1068-9613.

M. A. Patterson, M. Weinstein, and A. V. Rao. An efficient overloaded method forcomputing derivatives of mathematical functions in MATLAB. ACM Trans.Math. Softw., 39(3):17:1–17:36, may 2013. ISSN 0098-3500. doi:10.1145/2450153.2450155. URLhttp://doi.acm.org/10.1145/2450153.2450155.

42/ 44

References IV

L. Rich and D. Hill. Automatic differentiation in MATLAB. Applied NumericalMathematics, 9(1):33–43, 1992. ISSN 0168-9274. doi:10.1016/0168-9274(92)90065-L.

J. Shin and P. Hovland. Comparison of two activity analyses for automaticdifferentiation: Context-sensitive flow-insensitive vs. context-insensitiveflow-sensitive. Proceedings of the ACM Symposium on Applied Computing,pages 1323–1329, 2007. doi: 10.1145/1244002.1244287.

A. Verma. ADMAT: Automatic Differentiation in MATLAB Using Object OrientedMethods. In M. E. Henderson, C. R. Anderson, and S. L. Lyons, editors, ObjectOriented Methods for Interoperable Scientific and Engineering Computing:Proceedings of the 1998 SIAM Workshop, pages 174–183. SIAM,Philadelphia, 1999.

A. Walther and A. Griewank. ADOL-C: Computing higher-order derivatives andsparsity pattern for functions written in C/C++. 2004.

43/ 44

References V

M. J. Weinstein and A. V. Rao. ADiGator, a Toolbox for the AlgorithmicDifferentiation of Mathematical Functions in MATLAB Using SourceTransformation via Operator Overloading. ACM Transactions on MathematicalSoftware, Submitted, 2015. URL http://vdol.mae.ufl.edu/

SubmittedJournalPublications/adigator-CALGO.pdf.

44/ 44