C t ti l St t i f LC t ti l St t i f L S lS lComputational Strategies for LargeComputational Strategies for Large--Scale Scale Estimation in Dynamic SystemsEstimation in Dynamic Systems

Victor M. ZavalaPostdoctoral ResearcherPostdoctoral Researcher

Mathematics and Computer Science DivisionArgonne National Laboratory

vzavala@mcs.anl.gov@ g

Joint Work with: Larry Biegler (Carnegie Mellon), Carl Laird (Texas A&M)

Eindhoven, NetherlandsJune 9th, 2009

MotivationMotivationDegrees of

Maximum Likelihood (Multi-Set)

gFreedom

DAEs (Physical Model)

Inequalities (Physical Limits)

I i i l C di i

Output Mapping (Measured)

Initial Conditions

Objective: - Detailed Models (>1,000 DAEs), Many Experimental Data SetsObjective: Detailed Models ( 1,000 DAEs), Many Experimental Data Sets - Errors-in-Variables, Inference Analysis- Fast Solutions (Feedback Control, Profiling Methods)

Challenge: - Degrees of Freedom, Computational Expensive DAEsNonlinearity Ill Posedness (Observability)- Nonlinearity, Ill-Posedness (Observability)

OutlineOutline

Message: Message: Current Optimization Tools Suitable for Current Optimization Tools Suitable for LargeLarge--ScaleScale Estimation TasksEstimation Tasks

1.1. Nonlinear Programming (NLP) FrameworkNonlinear Programming (NLP) FrameworkDiscretization and Interior Point SolversDiscretization and Interior Point SolversInternal Regularization and ObservabilityInternal Regularization and Observability

2.2. Inference Analysis and Parallel DecompositionInference Analysis and Parallel DecompositionE l i i Li Al bE l i i Li Al bExploiting Linear AlgebraExploiting Linear Algebra

3. Industrial Case Study3. Industrial Case StudyPolyethylene ReactorPolyethylene Reactor

4. Conclusions and Future Work4. Conclusions and Future Work

1.1. Nonlinear Programming FrameworkNonlinear Programming Framework

NLP FrameworkNLP Framework

Traditional Estimation Tools: Simulation-Based, Gauss-Newton (Avoid Second Derivatives)

Limitations: Degrees of Freedom, Boundary Conditions, Unstable Systems

Full Discretization + Sparse NLP Solvers

NLP Discretized DAE Model

Large-Scale and Sparse NLP

All-At-Once Solution (Collocation, Splines, Galerkin)

Cheap Exact First and Second Derivatives - Automatic Differentiation (AMPL, ADOL-C)

NLP FrameworkNLP FrameworkIPOPT Wächter, Biegler 2006, g

NLP Barrier Problem

Solve Sequence of BPs with

KKT MatrixNewton’s Method

KKT Matrix

Advantages of General NLP Solvers Favorable Complexity vs. I li i S d D i iInequalities, Second Derivatives (Number of Iterations Not Affected by Degrees of Freedom)Crucial to Exploit Sparsity (Reduce Time and Memory Requirements)

NLP FrameworkNLP Framework

KKT Matrix

- General Sparse Linear Solvers -MA57 from Harwell-Phase I: Reordering to Preserve Sparsity Phase II: Numerical Factorization

O i i l R d dOriginal Reordered

Phase IIPhase I

(Default) Approximate Minimum-Degree Reordering Duff, 1993

Local Analysis – Fails at Identifying High-Level Structures Elimination Tree Davis, 2006

Nested Dissection Reordering Gould, 2004, Karypis, 1999

Global Analysis - Identifies High-Level Structuresy g

Efficient for DAEs and PDEs Discretizations

2. Inference Analysis and Parallel Decomposition2. Inference Analysis and Parallel Decomposition

Inference AnalysisInference Analysis

Observability B & B l 2000

Limitation of General NLP Solvers Solution Analysis Difficult: Are Parameters Observable? How to Get Posterior Covariance?

Observability Basu & Bresler, 2000

- Parameters Observable IF Can be Infered Uniquely from Measurements- Parameters Observable IF NLP Has a Unique Local Minimizer

UniqueNon-Unique

Context of NLP Solver- Internal Regularization of Hessian in Presence of Nonconvexity (as Levenberg-Marquardt)

Strong Second-Order Conditions Nocedal & Wright, 1999- at Solution Implies Local Isolated Minimizer (Needs Second Derivatives!)

Inference AnalysisInference AnalysisD f F d (P t )Consider Formulation Degrees of Freedom (Parameters)Consider Formulation

Reduced Hessian Hessian Projected onto Null Space of Constraints

Approximation of Parameters Posterior Covariance Bard, 1974, Generalized

j p

Uncertainty Levels

K*Already Factorized

Theorem: Zavala & Biegler, 2007

Large-Scale Inference Analysis Possible using KKT Matrix (Available AMPL and C++ Interface)

Toy ProblemToy ProblemFit Third Order Polynomial to Increasing Number of MeasurementsFit Third Order Polynomial to Increasing Number of Measurements

- Unique solution only for , Obtain Confidence Regions from IPOPT

AMPL Model

Toy ProblemToy ProblemIPOPT OutputIPOPT Output

Confidence Regions

Parallel DecompositionParallel DecompositionMulti-Set Estimation Problem

Parameters Co ple ProblemParameters Couple Problem

General Optimization

Multi-ScenarioOptimization

Direct Factorization Tailored Factorization, Laird & Biegler 2006

Reaches Memory BottleneckFactorization Time Increases (Linearly)

,Distribute Over Parallel Processors

Factorization Time Constant

3. Industrial Case Study3. Industrial Case Study

LowLow--Density PolyethyleneDensity PolyethyleneReactor

Temperat resJacket

TemperaturesI iti t I i i I iti t I iti t Temperatures TemperaturesInitiators Initiators Initiators Initiators

Flowrate EthyleneInlet Temperatures

Low-Pressure Recycle

Ethylene Cold-Shots

Recycle Systemand Flash SeparationHigh-Pressure Recycle

LDPEPolymer Melt Index

Chain-Transfer Agent

- Polymerization at High Pressures (2000 atm, Complex Transport and Kinetic Effects)

- Persistent Fouling (Spatio-Temporal Disturbance)

Wid O ti R i ( > 20 G d )- Wide Operating Regions ( > 20 Grades)Jacket

Core

Wall

Initiator(s) Initiator(s)

LowLow--Density PolyethyleneDensity Polyethylene

Cw Cw

NZ

CwInitiator(s) ( )

Reaction Cooling Reaction

1 2 NZ

C

EthyleneLDPE

Cw Cw Cw

Conservation Mass, Energy, Momentum

Thermodynamic & Transport Properties

Initial Conditions

Output Mapping

Boundary Conditions

Initial Conditions

Large Set of Partial Differential and Algebraic Equations (PDAEs)Discretize in Space - 10,000 DAEs in Time

Polymerization Kinetic Mechanism

LowLow--Density PolyethyleneDensity Polyethyleney

~ 35 Elementary ReactionsKinetic ParametersUnder Controlled

Laboratory Conditions

Estimation with Industrial Process Data

LowLow--Density PolyethyleneDensity PolyethyleneEstimation with Industrial Process Data

Cw Cw CwInitiator(s) Initiator(s)

Temperatures

1 2 NZEthylene

LDPE/Gas

Reaction Cooling Reactionp

Cw Cw Cw

CTA

Polymer Properties

(Laboratory)Comonomer

(Laboratory)

F I d i l C ll b (S d S D S )Output Measurements (Temperatures at Discrete Positions, Laboratory Polymer Properties)

Input Measurements (Inlet Temperatures, Flow rates, etc.)

From Industrial Collaborator (Steady-State Data Sets)

p ( p , , )

R fi Ki i P d Di b ( H T f C ffi i )Off-Line Estimation Objective

Refit Kinetic Parameters and Disturbances (e.g. Heat-Transfer Coefficients)

Off Line Estimation Problem

LowLow--Density PolyethyleneDensity PolyethyleneOff-Line Estimation Problem

Least-Squares Objective

Steady-StateReactor Model

1) Standard Least-Squares (SLS) - Parameters (Do not Correct Noisy Inputs) -Biased-

2) Errors-in-Variables Measured (EVM) - Parameters (Correct Noisy Inputs) –UnBiased-

1 Data Set 10 Data Sets10

NLP Size with Number of Data Sets

x 1010,000 Constraints 100,000 Constraints

Large Number of Degrees of FreedomLarge Number of Degrees of FreedomOrder of 100 (SLS) to 1,000 (EVM)

LowLow--Density PolyethyleneDensity Polyethylene

Fit of Axial Temperature Profile

Fit of Polymer Properties Zavala & Biegler, 2006

LowLow--Density PolyethyleneDensity Polyethyleney p g ,

Is ModelIs Model Structure Correct?

Average Deviation 14 Different Grades -6 Estimation, 8 Trial-

Inference Analysis

LowLow--Density PolyethyleneDensity Polyethyleney

Most Parameters Estimated Reliably With Few Sets (Except Pressure-Dependent)

Computational Results – IPOPT & Default Linear Algebra (Pentium IV PC)

LowLow--Density PolyethyleneDensity Polyethylene

1) One Data Set (SLS)

Computational Results IPOPT & Default Linear Algebra (Pentium IV PC)

2) Multiple Data Sets (SLS)

8 i~ 8 min~ 15 min

3) Multiple Data Sets (EVM)

~ 16 min~ 15 min 16 min

Solution Strategy is EfficientgyBut, Memory Requirements Can Become Bottleneck in Very Large Problems

IPOPT with Parallel Linear Algebra Zavala, Laird & Biegler 2007

LowLow--Density PolyethyleneDensity Polyethyleneg , g

Beowulf Cluster, Pentium IV, 3.0 GHz Processors

Default ApproachOut of Memory

30 Minutes

< 10 Minutes

340,000 Equations2,100 Degrees of Freedom, g

Parallelization Avoids Memory Limitations

On Line Estimation Problem (MHE)

LowLow--Density PolyethyleneDensity PolyethyleneOn-Line Estimation Problem (MHE)

Least-Squares Objective

D iDynamicReactor Model

Inputs and Temperature Measurements Available On-Line (No Laboratory Data)

Closed-Loop Inference of Model States and Parameters using Moving Measurements Window

Jacket

Core

Wall (Not Measured)

Core

Solution Time Critical (Difficult Parallelization), Uncertainty Required

Fouling-Defouling with Synthetic Data

LowLow--Density PolyethyleneDensity Polyethyleneg g y

Fouling

Cleaning

TimeMeasured

TimeCoreTemperature

Time

MeasuredJacket

Temperature

Reconstruction of Wall Profile

LowLow--Density PolyethyleneDensity Polyethylene

Time

Reconstruction of Wall Profile

Wall Temperature

Initial Guess

Covariance Matrix of Axial Wall Profile

Uncertainty Levels

Steady-State Covariance Matrix Reached After Few Time Steps

Computational Results (IPOPT with Nested Dissection on Pentium IV PC)

LowLow--Density PolyethyleneDensity Polyethylenep

NLP ~ 80,000 Constraints, 370 Degrees of Freedom

Sampling Time = 2 min

in]

Tim

e [m

iC

PU

Effect of KKT Matrix Reordering on Solution Time

c] Nested DissectionMinimum Degree

Tim

e [s

ecC

PU T

Dissection ~ 350,000 Constraints and 1,000 Degrees of Freedom ~ 2 Minutes (Warm-Start)

4. Conclusions and Future Work4. Conclusions and Future Work

C t O ti i ti T h l S it bl f L S l E ti ti

Conclusions and Future WorkConclusions and Future WorkCurrent Optimization Technology Suitable for Large-Scale Estimation

Automatic Differentiation, Sparse Linear Algebra

General Modeling and Optimization Tools - Try!AMPL for Modeling and Derivatives (Commercial but Cheap)ADOL-C for Exact Derivatives (Open-Source) ( p )IPOPT for Solution and Inference (Open-Source)

F t W kFuture WorkStrategies for 4-D PDE SystemsLithium-Ion Batteries (Complex Aging Phenomena, Systematic Estimation Needed)

C t ti l St t i f LC t ti l St t i f L S lS lComputational Strategies for LargeComputational Strategies for Large--Scale Scale Estimation in Dynamic SystemsEstimation in Dynamic Systems

Victor M. ZavalaPostdoctoral ResearcherPostdoctoral Researcher

Mathematics and Computer Science DivisionArgonne National Laboratory

vzavala@mcs.anl.gov@ g

Joint Work with: Larry Biegler (Carnegie Mellon), Carl Laird (Texas A&M)

Eindhoven, NetherlandsJune 9th, 2009