1

4. Model constraints

Quimiometria Teórica e AplicadaInstituto de Química - UNICAMP

2

Principal component analysis (PCA)Principal component analysis (PCA)

• In Hotelling’s (1933) approach, components have maximum variance.– X = TPT + E – Components are calculated successively.– Components are orthogonal: TTT = Diagonal; PTP = I

• In Pearson’s (1901) and Eckart & Young’s (1936) approach, components explain maximum amount of variance in the variables.– X = ABT + E – Components are calculated simultaneously.– Components have no orthogonality or unit-length

constraints.

3

Constrained least squaresConstrained least squares

• Solve under the constraint that A is non-negative, unimodal, smooth etc.

2Tmin ABXA

• Some constraints are inactive, e.g. PCA under orthogonality.

• If constraints are active, A is no longer the least-squares solution.

4

Why use constraints?Why use constraints?

• Obtain solutions that correspond to known chemistry, making the model more interpretable.– Concentrations can not be negative.

• Obtain models that are uniquely identified.– Remove rotational ambiguity.

• Avoid numerical problems such as local minima and swamps.– Constraints can help ALS find the correct solution

5

Example: curve resolution of HPLC data (1)Example: curve resolution of HPLC data (1)

• HPLC analysis of three coeluting organophosphorus pesticides.

• Diode-array detector gives a spectrum at each time point: X (time wavelength).

• Beer-Lambert law says X = CST + E.

• Initial analysis shows that three analytes are present. Data is from Roma Tauler’s web-site http://www.ub.es/gesq/eq1_eng.htm

Download it and try for yourself!

6

Example: curve resolution of HPLC data (2)Example: curve resolution of HPLC data (2)Unconstrained solutionUnconstrained solution

99.990094% of X explained

Calculation time: 0.43 seconds

31 31.1 31.2 31.3 31.4 31.5 31.6 31.7-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

Elution time (min)

Con

cent

ratio

n (u

nit)

180 200 220 240 260 280 300 320 340 360 380-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

Wavelength (nm)A

bsor

btio

n (u

nit)

C S

7

Example: curve resolution of HPLC data (2)Example: curve resolution of HPLC data (2)Non-negativity constraintsNon-negativity constraints

31 31.1 31.2 31.3 31.4 31.5 31.6 31.70

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Elution time (min)

Con

cent

ratio

n (u

nit)

180 200 220 240 260 280 300 320 340 360 3800

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

Wavelength (nm)A

bsor

btio

n (u

nit)

99.990079% of X explained

Calculation time: 16 seconds

C S

8

Example: curve resolution of HPLC data (3)Example: curve resolution of HPLC data (3)Unimodality & non-negativity constraintsUnimodality & non-negativity constraints

99.989364% of X explained

Calculation time: 16 minutes

31 31.1 31.2 31.3 31.4 31.5 31.6 31.70

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Elution time (min)

Con

cent

ratio

n (u

nit)

180 200 220 240 260 280 300 320 340 360 3800

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

Wavelength (nm)A

bsor

btio

n (u

nit)

C S

9

CommentsComments

• Active constraints always reduce % fit, but can give a more interpretable model.

E3X3 C3

E2X2 C2

• It is possible to ‘stack’ two–way data from different experiments, e.g.

= +

E1X1 C1

S

10

What sort of constraints might be useful?What sort of constraints might be useful?

• Hard target: known spectrum, ar = s• Non-negativity: concentrations, absorbances• Monotonicity: kinetic profiles• Unimodality: elution profiles, fluorescence excitations• Other curve shapes: Gaussian peaks, symmetry• Selectivity: pure variables• Functional constraints: first-principle models• Closure: [A]t + [B]t + [C]t = y

• Orthogonality: useful for separation of variances• ...plus many more...

11

Conclusions (1)Conclusions (1)

• It is possible to mix constraints within the same mode, i.e. loadings 1 and 3 are non-negative, loadings 2 are unimodel.

• Chemical knowledge can be included in your model by using constraints.

• Constraints can improve the model making it– closer to reality– easier to understand– more robust to extrapolation

12

Conclusions (2)Conclusions (2)

• Mixed constraints can be applied using column-wise estimation:1. Subtract contribution from other components

2. Estimate component under desired constraint2Tmin rr

r

r

baXa

– Bro & Sidiropoulos (1998) have shown that this is equivalent to solving

where is the unconstrained solution,

2min T

aarr

r

rrrr

r bbbX T

Trrr BAXX

13

ALS for Tucker3ALS for Tucker3• Step 0: Initialise B, C & G

2TJKI

TTRRRT

min

321

AZX

BCGZ

A

• Step 1: Estimate A:

2

TTT

Gvecvecmin ZGX

ABCZ

• Step 4: Estimate G:

• Step 5: Check for convergence. If not, go to Step 1.

• Step 3: Estimate C in same way:2TIJKmin CZX

C

• Step 2: Estimate B in same way:2TKIJmin BZX

B

14

Example: UV-Vis monitoring of a chemical reaction (1)Example: UV-Vis monitoring of a chemical reaction (1)

• Two-step conversion reaction under pseudo-first-order kinetics:

A + B C D + E

• UV-Vis spectrum (300-500nm) measured every 10 seconds for 45 minutes

• 30 normal batches measured: X (30 201 271)

• 9 disturbed batches: pH changes made during the reaction

15

Example: UV-Vis monitoring of a chemical reaction (2)Example: UV-Vis monitoring of a chemical reaction (2)3-component PARAFAC model has problems!3-component PARAFAC model has problems!

1 27-5

0

5Batch mode

Load

ing

1

1 27-5

0

5Lo

adin

g 2

1 27-0.5

0

0.5

Load

ing

3

Batch number

300 500-0.2

0

0.2Wavelength mode

300 500-0.2

0

0.2

300 500-0.2

0

0.2

Wavelength

0 450.085

0.09

0.095Time mode

0 450.06

0.08

0.1

0.12

0 45-0.5

0

0.5

Time

spectra are difficult to interpret

highly correlated

16

Example: UV-Vis monitoring of a chemical reaction (3)Example: UV-Vis monitoring of a chemical reaction (3)External process informationExternal process information

300 320 340 360 380 400 420 440 460 480 5000

0.2

0.4

0.6

0.8

1

1.2

1.4

Wavelength (nm)

Abs

orba

nce

(uni

ts)

300 320 340 360 380 400 420 440 460 480 5000

0.2

0.4

0.6

0.8

1

1.2

1.4

Wavelength (nm)

Abs

orba

nce

(uni

ts)

A

D

No compound interactions allowed: Lambert-Beer law

First-order reaction kinetics are known:

Pure spectra of reactant and product known:

ttt

tktkt

tkt

eekk

ke

CAAD

AC

AA

0

12

01

0

21

1

17

Example: UV-Vis monitoring of a chemical reaction (4)Example: UV-Vis monitoring of a chemical reaction (4)Constrained Tucker3 (1,3,3) modelConstrained Tucker3 (1,3,3) model

B

C

G=

wavelength

time

batch

A

+ E

REACTION KINETICS

KNOWN SPECTRALAMBERT-

BEER LAW

X

X = AG (CB)T + E

18

Example: UV-Vis monitoring of a chemical reaction (5)Example: UV-Vis monitoring of a chemical reaction (5)Constrained Tucker3 (1,3,3) modelConstrained Tucker3 (1,3,3) model

• Core array: G = [g111 0 0 | 0 g122 0 | 0 0 g133]

1 27-0.5

0

0.5Batch mode

Load

ing

1

Batch number300 5000

0.1

0.2Wavelength mode

300 5000

0.1

0.2

Load

ing

2

300 5000

0.1

0.2

Load

ing

3

Wavelength

0 450

0.5

1Time mode

0 450

0.5

1

0 450

0.5

1

Time

*

*

*

*

*

fixed to known spectrum

fixed to 1st-order kinetics

Rate constants are found! k1 = 0.27, k2 = 0.029

Spectrum of intermediate is found!

19

Conclusions (3)Conclusions (3)

• It is possible to build ‘hybrid’ or ‘grey’ models where some loadings are constrained and others are left free – see the extra material which follows!

• If you already have some information about your chemical process, then include it in your model

• Using constraints can really help to uncover new information about your data (e.g. find spectra, estimate rate constants, test models).

20

Extra material: Black vs white modelsExtra material: Black vs white models

• ‘Black-box’ or ‘soft’ models are empirical models which aim to fit the data as well as possible e.g. PCA, neural networks

• ‘White’ or ‘hard’ models use known external knowledge of the process e.g. physicochemical model, mass-energy balances

Difficult to interpret

Good fit

Easy to interpret

Not always availableGood fit

• ‘Grey’ or ‘hybrid’ models combine the two.

+

21

Extra material: Grey models mix black and white modelsExtra material: Grey models mix black and white models

++Total variation

Systematic variation due

to known causes

Systematic variation due to unknown

causes

Unsystem-atic variation

RESIDUALSMODEL

REACTION KINETICS

MECHANISTIC MODEL

KNOWN CONCENTRATIONS

22

Extra material: Grey modelExtra material: Grey model

B

C

G +=

wavelength

time

batch

A

B

C

G + E

REACTION KINETICS

KNOWN SPECTRALAMBERT-

BEER LAW

X

A

23

Extra material: Grey model parameter estimationExtra material: Grey model parameter estimation

White part Black part

A - Ordinary least squares[a1 a2 a3]

BFixed (target) loadings

b1 = reactantb3 = product

Ordinary least squares[b2 b4 b5]

CFirst-order kinetic model

Levenberg-Marquardtoptimisation for

[c1 c2 c3] = f(k1,k2)

Ordinary least squares[c4 c5]

GRestricted core array

Non-interacting triads have gpqr =0 according to Lambert-Beer

Ordinary least squares(vectorised)G for gpqr 0

24

Extra material: Grey model parametersExtra material: Grey model parameters

White components Black components describe known effects can be interpreted

• 99.8% fit (corresponds well with estimated level of spectral noise of 0.13%)

1 27-0.5

0

0.5Batch mode

Load

ing

1

Batch number300 5000

0.1

0.2Wavelength mode

300 5000

0.1

0.2

Load

ing

2

300 5000

0.1

0.2

Load

ing

3

Wavelength

0 450

0.5

1Time mode

0 450

0.5

1

0 450

0.5

1

Time

1 27-0.4-0.2

00.20.4

Batch mode

Load

ing

1

1 27-0.4

-0.2

0

0.2

Load

ing

2

Batch number

300 500-0.1

0

0.1

Wavelength mode

300 5000

0.1

0.2

Wavelength

0 45-0.1

0

0.1

0.2

Time mode

0 450.084

0.086

0.088

0.09

Time

*

*

*

*

*

25

Extra material: Grey model residualsExtra material: Grey model residuals

0 10 200

0.005

0.01

0.015

0.02

Batch number

Squ

ared

resi

dual

s

300 350 400 450 5000

1

2

3

4

5x 10

-3

Wavelength

Squ

ared

resi

dual

s

0 5 10 15 20 25 30 35 40 450

0.002

0.004

0.006

0.008

0.01

Time

Squ

ared

resi

dual

s

26

Extra material: Off-line monitoringExtra material: Off-line monitoring

0 5 10 15 20 25 30 35 400

5

10

15

20

25

30

35

33

34

38

Batch number

D-s

tatis

tic

Off-line monitoring: D-statistic with 95% and 99% confidence limits

0 5 10 15 20 25 30 35 400

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

31

32

33

3435

36

37

38

39

Batch number

Q-s

tatis

tic

Off-line monitoring: Q-statistic with 95% and 99% confidence limits

D-statistic Q-statistic

(within model variation) (residual variation)

27

Extra material: On-line monitoring of disturbed batchExtra material: On-line monitoring of disturbed batch

0 5 10 15 20 25 30 35 40 450

5

10

15

20

Time

D-S

tatis

tic

On-line monitoring: D-statistic with 95% and 99% confidence limits

0 5 10 15 20 25 30 35 40 45-9

-8

-7

-6

-5

-4

Time

ln(S

PE

)

On-line monitoring: SPE with 95% and 99% confidence limits