Stochastic Collocation Methods for Polynomial Chaos...
Transcript of Stochastic Collocation Methods for Polynomial Chaos...
Stochastic Collocation Methods for Polynomial Chaos: Analysis and Applications
Dongbin
Xiu
Department of Mathematics, Purdue University
Support:
AFOSR FA9550-08-1-0353 (Computational Math)NSF CAREER DMS-0645035 (Computational Math)DOE DE-FC52-08NA28617 (PSAAP)
Overview
• Generalized polynomial chaos
• Stochastic collocation• Lagrange interpolation• Pseudo spectral gPC
• Applications• Bayesian inverse problem• Data assimilation
0
ˆ( , , ) ( , ) ( ), # of basisN
zN
z
n Nu t x Z u t x Z
nk kk=
⎛ ⎞+ ⎟⎜ ⎟Φ = ⎜ ⎟⎜ ⎟⎝ ⎠∑• Nth-order gPC
expansion:
( ) ( ) ( ) ( ) ( )Z Z Z Z Z Z dZi j i j ijE ρ δ⎡ ⎤Φ Φ Φ Φ =⎣ ⎦ ∫• Orthogonal basis:
Uncertainty Quantification via gPC
0
( , , ) ( ), (0, ]
( ) 0, [0, ]( , ), { 0}
z
z
z
n
n
n
u t x Z u T Dtu T D
u u x Z t D
∂⎧ = × ×⎪ ∂⎪
= ×∂ ×⎨⎪ = = × ×⎪⎩
L
B
R
RR
• Stochastic PDE:
ˆ [ ( ) ( )] ( ) ( ) ( ) , 0 ,u u Z Z u Z Z Z dZ Nk k k kE ρ= Φ = Φ ≤ ≤∫
2 2( ) ( )inf
nzN
N L Z L Zu u u
ρ ρΨ∈Π− = − Ψ• Optimality:
Generalized Polynomial Chaos (gPC)
• Implementations:Stochastic GalerkinStochastic collocation
• Properties:Rigorous mathematicsHigh accuracy, fast covergenceCurse-of-dimensionality
• Basis functions:Hermite polynomials: seminal work by R. GhanemGlobal orthogonal polynomials (Xiu & Karniadakis, 02)Wavelet basis (Le Maitre et al, 04)Piecewise basis (Babuska et al 04, Wan & Karniadakis, 05)
• Lagrange interpolation• Can not be constructed for any given nodes• Interpolation error hard to control
• Pseudo spectral• Utilize gPC
polynomial basis
• Becomes multivariate integration
Stochastic Collocation
• Sampling:
(solution statistics only)• Random (Monte Carlo)• Deterministic (lattice rule, tensor grid, cubature)
• Response surface method• Multivariate interpolation• Many ad hoc approaches
• Collocation:
To satisfy governing equations at nodes
• Stochastic collocation:
To construct polynomial approximations
Stochastic Collocation –
Lagrange Interpolation
1
( ) ( ) ( )Q
Q jj
j
u Z u Z L Z=∑ ( ) , 1 ,j
i ijL Z i j Qδ= ≤ ≤• Lagrange interpolation:
{ }1
zQ ni
Q iZ R
=Θ = ∈• Nodal set:
0
( , , ) ( ), in (0, ] ,
( ) 0, [0, ] ,( , ), { 0}
j
j
u t x Z u T Dtu T D
u u x Z t D
∂= ×
∂= ×∂
= = ×
L
B
• Solution:
for j=1,…,Q,
(Xiu & Hesthaven, SIAM J. Sci. Comput., 05)
( )1 nziiU U⊗ ⊗
• Tensor product:
1
1
1( 1) ( )nziq i
q N q
NU U
qi
i i−
− + ≤ ≤
⎛ ⎞− ⎟⎜ ⎟− ⋅ ⋅ ⊗ ⊗⎜ ⎟⎜ ⎟⎜ −⎝ ⎠∑• Sparse grid (Smolyak):
Stochastic Collocation: Pseudo Spectral Approach
• Nth-order gPC
projection
2 ( )Q N N L Zu w
ρε −• Aliasing Error:
0
ˆ( , , ) ( , ) ( ),N
Nw t x Z w t x Zk kk=
= Φ∑
1
ˆ ( , , ) ( ) ( ) ( ) ( )Q
j j j
j
w u t x Z Z u Z Z Z dZk k kα ρ=
= Φ ≈ Φ∑ ∫
• gPC-collocation approximation
0
ˆ( , , ) ( , ) ( ),N
Nu t x Z u t x Zk kk=
= Φ∑ ˆ ( ) ( ) ( ) .u u Z Z Z dZk k ρ= Φ∫
ˆ ˆ( , ) ( , ),w t x u t x Q→ →∞k k
(Xiu, Comm. Comput Phys, vol. 2, 07)
gPC-Collocation: Algorithm{ }
11. Choose a nodal set , in z
Q nj j
jZ Rα
=
1
3. Evaluate the approximate gPC expansion coefficient
ˆ ( , , ) ( ) , 0 ;Q
j j j
j
w u t x Z Z Nk k kα=
= Φ ≤ ≤∑th
1
4. Construct the -order gPC approximation
ˆ ( , , ) ( ).N
N
N
w t x Z w Zk kk=
= Φ∑
Post-process
Deterministic solver
0
2. Solve for each 1, , ,
( , , ) ( ), in (0, ] ,
( ) 0, [0, ] ,( , ), { 0}
j
j
j Qu t x Z u T Dtu T D
u u x Z t D
=∂
= ×∂
= ×∂
= = ×
L
B
( )2
1/ 22 2 2 2( )N N Q QL Z
u w M Cρ
ε ε ε εΔ− ≤ + +
Finite-term projection error aliasing error Numerical errorError ≤ + +
• Error bound (Xiu, 07):
Parameter Estimation: Bayesian Inverse Approach
0
( , , ) ( ), (0, ]
( ) 0, [0, ]( , ), { 0}
z
z
z
n
n
n
u t x Z u T Dtu T D
u u x Z t D
∂⎧ = × ×⎪ ∂⎪
= ×∂ ×⎨⎪ = = × ×⎪⎩
L
B
R
RR
• Stochastic PDE:
( , , ) : [0, ] uz nnu t x Z T D× ×R R• Solution:
1
( ) ( )zn
Z i ii
z zπ π=
=∏• Prior distribution:
• Estimation of the prior distributionRequires direct measurements of the parametersNo/not enough direct measurements? (Use experience/intuition …)How to take advantage of measurements of other variables?
( ) , is i.i.d.dnd G Z e e= + ∈R• Data:
Bayesian Inference
( | ) ( )( ) ( | )( | ) ( )
d d Z ZZ Z dd Z Z dZ
π ππ ππ π
=∫
1
( ) ( | ) ( ( ))d
i
n
e i ii
L Z d Z d G Zπ π=
= −∏• Likelihood function:
• Posterior distribution:
• Notes:Difficult to manipulateClassical sampling approaches can be time consuming (MCMC, etc)GPC (Galerkin) based approach: (Marzouk, Najm, Rahn, JCP, 07)
( ) ( )( )( ) ( )
d NN
N
L Z ZZL Z Z dZ
πππ
=∫
,1
( ) ( | ) ( ( ))d
i
n
N N e i N ii
L Z d Z d G Zπ π=
= −∏
• gPC
approximation:
• Properties:Allows direct sampling in term of Z with arbitrarily large samples(Virtually) no additional computational cost – forward problem solver onlyConvergence seems natural
2(0, ), i.i.d. Normale N σ I∼
( ) 11 2 1
2
( )( ) log( )zD z dzz
ππ π ππ∫
Convergence of gPC
Bayesian Inference
• Kullback-Leibler
divergence:
• Observation error:
( )
2
2
i ,
. If the gPC expansion converges to in , then the posterior density
converges to in the sense
D 0, .
Moreover, if
( ) ( )
Z
z
N
d dN
d dN
N i L
G G L
N
G Z G Zπ
π
π π
π π → →∞
− ≤
Theorem
( ) 2
, 1 , 0, independent of ,
then for sufficiently large ,
.
d
d dN
CN i n C N
N
D N
α
α
α
π π
−
−
≤ ≤ >
∼
• Notes:Fast (exponential) convergence rate is retainedFactor of 2 in the convergence rates
(Marzouk & Xiu, Comm. Comput. Phys. 08)
Parameter Estimation: Supersensitivity
Example
[ ]2
2 , 1,1u u uu xt x x
ν∂ ∂ ∂+ = ∈ −∂ ∂ ∂
• Burgers’
equation :
( 1) 1 ( ); (1) 1; 0 1u Z uδ δ− = + =− < <<• Boundary conditions :
(Xiu & Karniadakis, Int. J. Numer. Eng., 04)1% uncertainty in left BC
Deterministic results with no uncertainty
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
5
10
15
20
25
δ
p(δ
| dat
a )
exact posteriorgPC, p=4gPC, p=8δ
true
Measurement noise: e~N(0,0.052)
Prior distribution is uniform
6 7 8 9 10 11 12 13 14 15 16
10−3
10−2
N
erro
r
D(πN
|| π )
||G − GN
||L
2
Factor = 2.10 (theory = 2)
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−0.2
0
0.2
0.4
0.6
0.8
1
1.2
z
G(z
) or
GN
(z)
exact forward solutiongPC approximation
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10
0.5
1
1.5
2
2.5
3
z
πd (z)
exact posteriorgPC posterior
Parameter Estimation: Step Function
• Assume the forward model is a step function• Posterior distribution is discontinuous• Gibb’s oscillations exist• Slow convergence with global gPC
basis functions
Forward model and its approximation Posterior distribution and its approximation
2 4 6 8 10 12 14 16 18 20
10−0.9
10−0.8
10−0.7
10−0.6
10−0.5
10−0.4
10−0.3
10−0.2
10−0.1
p
erro
r
D(π
N || π )
||G − GN
||L
2
Factor = 1.99 (theory = 2)
Kalman
Filter for Data Assimilation
• Forecast:
• Observation:
• Analysis:
0
( , ) ( , ), (0, ]
(0, )
ff
f
d t Z t t Tdt
Z
⎧= ∈⎪
⎨⎪ =⎩
u F u
u u
, :t m= + ∈ →d Hu ε HR R R
• True
state (unknown): , 1t m m∈ ≥u R
( )a f f= + −u u K d Hu
1( )f T f T −= +K P H HP H R
( )( )f f t f t T⎡ ⎤= − −⎣ ⎦P u u u uE T⎡ ⎤= ⎣ ⎦R εεE
(Kalman
gain matrix)
• Properties:• Straightforward for linear dynamic equations• Extension to nonlinear equations: Extended KF (EKF)• Optimal for Gaussian• Explicit calculation of covariance can be costly
Ensemble Kalman
Filter (EnKF)
( )( )f ff f f T f
e = − − ≈P u u u u P Te = ≈R εε R
( ) ( ) ( )( ) , 1, ,a f fe ii i i
i M⎡ ⎤= + − =⎣ ⎦u u K d H u …
1( )f T f Te e e e
−= +K P H HP Η R
• Properties:nonlinear dynamicssampling errors
Measurement. Can be eliminated by square-root filter (EnSRF)Solution states.
Computational cost is of great concern
( ) ( , ), 1, ,f f i
it Z i M=u u … ( ) ( ) , 1, ,ii
i N= + =d d ε …• Ensemble:
Error Analysis of the EnKF
1n nT t t+Δ = −
( )1
,
fn t t
p
e
O t N α
ε ε ε
σ
+ Δ Δ
−
≤ ⋅ + Δ ⋅ + Δ ⋅ ⋅
Δ
M K K H
∼
= −M I KH eΔ = −K K K
( )01
expn
n k nk
E E e t=
⎛ ⎞≤ + Λ ⋅⎜ ⎟⎝ ⎠
∑
• Lemma
(local error):
• Theorem
(global error):
1T −Λ ∝ Δ
• Note the inverse dependence
on assimilation step size
• Assimilation step size:
(Li & Xiu, vol. 197, CMAME 08)
EnKF
Example: Linear Wave Equation
0 100 200 300 400 500 600 700 800 900 10000
5
10
0 100 200 300 400 500 600 700 800 900 10000
5
10
0 100 200 300 400 500 600 700 800 900 10000
2
4
6
8
Long-term Wave propagation
t=5
t=100
t=500
50( , )x Z ∈ ×R R
Model description:Linear advection equation;Periodic domain of length L=1000;Wave speed = 1; grid spacing=1; time step = 1;True states are sampled from a Gaussian process, with zero mean and unit variance, and a spatial de-correlation length of 20. The dimension of random space is nz=50.Four measurements uniformly in space are made every 5 time units.Measurement variance is 0.01.No model error.
Error Behavior of EnKF
4.5 5 5.5 6 6.5 7 7.5 8−8.5
−8
−7.5
−7
−6.5
−6
−5.5
−5
log (N)
log
(Err
or)
T=100T=500
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50
0.005
0.01
0.015
Standard Deviation of Measurements
Err
or
T=1000
T=500
w.r.t. ensemble size w.r.t. data noise level
0 0.5 1 1.5 2 2.5 3 3.5 4−11.5
−11
−10.5
−10
−9.5
−9
−8.5
−8
−7.5
−7
−6.5
log(Δ T)
log(
Err
or)
EnKFEnSRFqEnSRF−2qEnSRF−3
EnKF
Error Behavior
qEnSRF: EnSRF
combined with deterministic sampling using optimal cubature
GPC Collocation based Kalman
Filter
T=100 T=500 T=1,000 T=1,500
EnKF
(N=100) 5.3×10-3 3.4×10-3 2.3×10-3 1.7×10-3
EnKF
(N=103) 1.5×10-3 1.0×10-3 7.4×10-4 6.2×10-4
EnKF
(N=104) 4.6×10-4 2.8×10-4 2.4×10-4 1.9×10-4
gPC-KF (N=51) 3.5×10-4 1.6×10-4 1.1×10-4 9.0×10-5
gPC-KF (N=100) 1.8×10-4 7.9×10-5 5.6×10-5 4.6×10-5
Errors of assimilated results
• 50 dimensional random space
(Li & Xiu, vol. 197, CMAME 08)
Accuracy Improvement of EnKF
via gPC
• Use cubature –
equally weighted
• Use pseudo-spectral gPC
0
ˆ( , ) ( ) ( ),N
f fN t Z t Zk k
k
u u=
Φ∑
( )0
ˆ ˆ ˆ,Tf f f f
N NN< ≤
⎡ ⎤= = ⎢ ⎥⎣ ⎦∑0 k kk
u u P u u
Analytical expression in Z
Statistics
(Li & Xiu, J. Comput. Phys. 08)
( ) ( )0
ˆ ( ) , 1, , , 1N
f f iN ki
t Z i M M=
= Φ =∑ kk
u u …
( ) ( ) ( ) ( )' ', , 1, ,f f f a a a
N N N N N Ni i i ii M= + = + =u u u u u u …
( ) ,a f fN N N N= + −u u K d Hu 1( )f T f T
N N N−= +K P H HP Η R
( ) ( ) ( )' ' ', 1, ,a f f
N N N Ni i ii M= + =u u K H u …
( ) ( )1 1Tf T f T f T
N N N N
− −⎛ ⎞= + + +⎜ ⎟⎝ ⎠
K P H HP H R HP H R R
GPC Based Ensemble Kalman
Filter
• Ensemble:
• Square-root update:
Mean state update:
Perturbation update:
01 , (0)f f
f fdu ur u u udt A
⎛ ⎞= − − =⎜ ⎟
⎝ ⎠
0 0.2 0.4 0.6 0.8 12
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
Time
True stateMeasurementEstimate
Example: Nonlinear Population Dynamics
1 2 3 4 5 6 7 8 9−11
−10
−9
−8
−7
−6
−5
−4
−3
Degree of the gPC expansion (N)
log(
Err
or)
MeanStandard deviation
4 5 6 7 8 9 10 11 12 13−12
−10
−8
−6
−4
−2
0
Number of quardrature points (Q)lo
g(E
rror
)
MeanStandard deviation
Error Convergence
N=8, Q=10 is sufficient
0 1 2 3 4 5 6−12
−10
−8
−6
−4
−2
0
log(Ensemble size)
log(
Err
or)
Mean (gPC EnSRF)Std (gPC EnSRF)Mean (EnSRF)Std (EnSRF)
Comparison: gPC-KF vs
EnKF
( )dx y xdtdy x y xzdtdz xy zdt
σ
ρ
β
⎧ = −⎪⎪⎪ = − −⎨⎪⎪ = −⎪⎩
0 0 0( , , ) (1.508870, 1.531271, 25.46091)x y z = −
10, 28, 8 / 3σ ρ β= = =
0 2 4 6 8 10 12 14 16 18 20−20
0
20
Time
X
0 2 4 6 8 10 12 14 16 18 20−50
0
50
Time
Y
0 2 4 6 8 10 12 14 16 18 200
20
40
60
Time
Z
Nonlinear System Example: Lorenz Equations
Small deviation in initial condition (0.001 in x0 ) causes large deviation
0 5 10 15 200
0.5
1
1.5
2
2.5
3
Time
||Est
imat
e−T
rue
stat
e||
N=2 (gPC EnSRF)N=3 (gPC EnSRF)EnSRF
Qualitative Comparison: gPC
EnKF
vs
EnKF
• GPC EnKF: Q=53=125• EnKF: ensemble size = 104
Summary
• Point selection is crucial for the efficacy of stochastic collocation
• GPC expansion is much more than a forward UQ method• Bayesian inverse (Marzouk & Xiu, Comm. Comput. Phys, 08)• Kalman
filter for data assimilation (Li & Xiu, CMAME 08; JCP 08)