Reverse Engineering Dynamical Systems From Time-Series ... › ~platte › apm598 ›...

Outline Brief Basis Pursuit Review Sparse Polynomial Identification ODE System Identification PDE System Identification

Reverse Engineering Dynamical Systems FromTime-Series Data Using `1-Minimization

Robert Thompson

ASU School of Mathematical & Statistical Sciences

November 2011


1 Brief Basis Pursuit Review

2 Sparse Polynomial Identification

3 ODE System Identification

4 PDE System Identification


Basis Pursuit

Find the sparsest vector that satisfies a set of constraints:

Basis Pursuit

minx‖x‖1 s.t. y = Ax (LP)

Find the sparsest vector that comes close to satisfying a set ofconstraints:

Relaxed Basis Pursuit

minx‖x‖1 s.t. ‖y −Ax‖2 < ε (SOCP)


Sparse Polynomial Identification: the problem

1 Given a polynomial:

f(t) =N∑

i=1

αiφi(t) t ∈ (−1, 1)

where f is K-sparse: # {i : αi 6= 0} = K < N ,and the αi unknown.

2 Measure:y = (f(t1), f(t2), . . . , f(tm))T

K < m < N

3 Determine x = (α1, . . . , αN ).


Sparse Polynomial Identification: Basis Pursuit

f(t1)f(t2)

...f(tm)

︸︷︷︸

y∈Rm

=

φ1(t1) φ2(t1) . . . φN (t1)φ1(t2) φ2(t2) . . . φN (t2)

...φ1(tm) φ2(tm) . . . φN (tm)

︸︷︷︸

A∈Rm×N

α1

α2......αN

︸︷︷︸x∈RN,

K−sparse

Apply Basis Pursuit:

minx‖x‖1 s.t. y = Ax → f =

∑Ni=1 αiφi(t)


“Sparse Identification”?

Interpolation: given N + 1 points, find the consistent polynomial ofdegree N .

Regression: given P � N points, force data to fit pre-selectedmodel of degree N .

“Sparse Identification”: given m� N points, find the sparsestmodel that fits the data.


Sparse Polynomial Identification in Natural Basis

φi(t) = ti−1

A is Vandermonde matrix.

Badly conditioned!

y =

f(t1)f(t2)

...f(tm)

=

t01 t11 . . . tN−1

1

t02 t12 . . . tN−12

...t0m t1m . . . tN−1

m

α1

α2......αN


Sparse Polynomial Identification Experiments

1 Select random K-sparse x ∈ RN

2 Measure:y = (f(t1), . . . , f(tm))T

tm uniformly distributed in (−1, 1).

3 Construct:

Aij =(tj−1i

), i ∈ (1 . . .m), j ∈ (1, N)

4 Find:

x∗ = minx′∈RN

‖x′‖ s.t. Ax′ = y

5 Measure accuracy of recovery: e = ‖x∗ − x‖26 e < µ = success


Discovering 2D Sparse Polynomials - Recovery Example

−1 0 1−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

← Recovered from 25 samples on(0, 1)

−1 0 1−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

x x*

--------------------

-0.0000 0

0.0000 0

0.0000 0

-0.0000 0

0.0000 0

-0.0000 0

-0.0000 0

-0.0000 0

-0.1168 -0.1168

1.2525 1.2525

0.0000 0

0.5675 0.5675

0.0000 0

-0.0000 0

... ...

0.0000 0

-0.0000 0

0.0000 0

-1.5505 -1.5505

0.4812 0.4812

-0.0000 0

0.0000 0

... ...


How Well Does It Work?

m/N

k/m

1d polynomial recovery for N = 36, uniform sampling

0.25 0.5 0.75 1

0.75

0.5

0.25

0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.2

0

Each pixel plots recovery rate after 50 experiments, dark → betterrecovery rate.Increasing m doesn’t make as much difference as might be hoped.


Increasing m doesn’t help

Many sets of columns of V are nearly dependent.

Many K-sparse vectors x ∈ RN become indistinguishable oncetransformed by A to y ∈ Rm, even with m = N .

Increasing m can’t solve this problem.

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1


Comparison With Chebyshev-Sparse Polynomials

Consider same scenario, but now using Chebyshev polynomials:

φi(t) = Ti(t) = cos((i− 1) arccos(t))

y =

f(t1)f(t2)

...

...f(tm)

=

T1(t1) T2(t1) . . . TN (t1)T1(t2) T2(t2) . . . TN (t2)

......

T1(tm) T2(tm) . . . TN (tm)

α1

α2......αN



Vandermonde

m/N

k/m

1d polynomial recovery for N = 36, Chebyshev sampling

0.25 0.5 0.75 1

0.75

0.5

0.25

0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.2

0

ChebyshevSparse 1d Chebyshev polynomial recovery, N = 36

m/Nk/m

0.25 0.5 0.75 1

0.75

0.5

0.25

0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.2

0

Using Chebyshev basis functions, we realize improvement as mincreases.


Increasing m helps

Columns of C are orthogonal.

All vectors y = Ax will be distinguishable if we use full C.

If we use less than full C, orthogonality is lost, some vectors start tobecome indistinguishable.

V

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

C

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1


The Role of the Nullspace of A

Theorem

2K-sparse vectors in nullspace of A⇒ exist K-sparse x that can’t berecovered by any algorithm.

Proof.

Let z = x1 − x2 be 2K-sparse ⇒ x1, x2 are K-sparse.Az = A(x1 − x2) = 0⇒ Ax1 = Ax2


2-D Sparse Polynomial Identification

What about 2-D polynomials?

In natural basis: f(t, u) =∑

i+j=0..Q

αijtiuj

(tm, um) according to some distribution in (−1, 1)× (−1, 1).

y =

f(t1, u1)f(t2, u2)

...

...f(tm, um)

=

1 t1 u1 t1u1 t21 u2

1 . . .1 t2 u2 t2u2 t22 u2

2 . . ....

...1 tm um tmum t2m u2

m . . .

α00

α10

α01

α11

α20...



−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

1

−15

−10

−5

0

5

10

← Recovered from 25 samples on(0, 1)× (0, 1)

0

5

10

15

20

25

0

5

10

15

20

25

−15

−10

−5

0

5

10

x x*

--------------------

0 -0.0000

0 0.0000

0 0.0000

0 0.0000

-9.1652 -9.1652

-3.2926 -3.2926

0 -0.0000

0 -0.0000

.

.

0 0.0000

0 -0.0000

2.4264 2.4264

0 -0.0000

0 -0.0000

.

.

0 0.0000

0 -0.0000

4.2605 4.2605

0 0.0000



2d polynomial recovery, N = 36

m/N

k/m

0.25 0.5 0.75 1

0.75

0.5

0.25

0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.2

0

Similar to 1-d results.

Again increasing m doesn’t change much.


Dimension vs Order

1-d: f(t) =∑N

i=0 xiti. Goes up to order N .

2-d: f(t) =∑

i+j=0..Q

xijtiuj . N = (Q+ 1)(Q+ 2)/2.

For Q = 7, N = 36.

Increasing dimension reduces required order of terms

m/N

k/m

1d polynomial recovery for N = 36, Chebyshev sampling

0.25 0.5 0.75 1

0.75

0.5

0.25

0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.2

0


m/N

k/m

0.25 0.5 0.75 1

0.75

0.5

0.25

0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.2

0


Discovering 3d Sparse Polynomials

Did someone say 3-D?

In natural basis: f(t, u, v) =∑

i+j+k=0..Q

αijktiujvk

y =

266664f(t1, u1, v1)f(t2, u2, v2)

.

.

.f(tm, um, vm)

377775 =

2666666664

1 t1 u1 v1 t1u1 t1v1 u1v1 . . .1 t2 u2 v2 t2u2 t2v2 u2v2 . . .

.

.

.

.

.

.

.

.

.

.

.

.1 tm um vm tmum tmvm umvm . . .

3777777775

2666666666664

α000α100α010α001α110α101α011

.

.

.

3777777777775



−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

1

−4

−2

0

2

4

6

8

← Recovered from 25 samples on(0, 1)× (0, 1)× (0, 1)

−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

1

−4

−2

0

2

4

6

8

x x*

--------------------

0 0.0000

0 -0.0000

.

.

0 0.0000

0 -0.0000

-1.2378 -1.2378

0 0.0000

0 -0.0000

0 -0.0000

0 -0.0000

0 0.0000

0 0.0000

0 0.0000

0 -0.0000

6.0409 6.0409

0 -0.0000

3.5132 3.5132

-1.3987 -1.3987

0 -0.0000

0 -0.0000

-2.3806 -2.3806




m/N

k/m

0.25 0.5 0.75 1

0.75

0.5

0.25

0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.2

0

m/Nk/m


0.25 0.5 0.75 1

0.75

0.5

0.25

0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

Similar to 2d case.


Summary of Sparse Polynomial Identification

Can recover very sparse polynomials with a handful of data points.

In comparison with more ideal bases, Vandermonde has limitationsthat preclude increasing K = number of non-zero entries.

For higher dimensions, can avoid use of higher order terms inVandermonde matrix, but improvement is marginal (if at all).


ODE System Identification: the idea

Measure an unknown dynamical system at m points in time.

Determine the system of equations governing the behavior, includingparameter values.

Originally proposed by Wang, Yang, Lai, Kovanis, and Grebogi.

Wen-Xu Wang, Rui Yang, Y.-C. Lai, Vassilios Kovanis, and CelsoGrebogi, ”Predicting catastrophes in nonlinear dynamical systems bycompressive sensing”, Physical Review Letters 106, 154101 (2011).

Similar to polynomial discovery, but A and y will come from systemmeasurements.


ODE System Identification - 1d

Autonomous 1d discrete dynamical map xn+1 = f(xn)

Assume f has form f(x) =N∑

i=0

αixi

Measure f(xnj ), f(xnj+1), j = 1..mTry to recover the αi. Sparse coefficient vector: f is “simple”.

Similar to 1d polynomial discovery setup.

Note: m now really means taking 2m points, because we must takexnj and xnj+1.

Each (xi, xi+1) pair is a single sample in this setup.

y =

xj1+1

xj2+1

...xjm+1

=

1 xj1 . . . xN

j1

1 xj2 . . . xNj2

...1 xjm

. . . xNjm

α1

α2

...

...αN


Example - Logistic Map

xn+1 = f(xn) = rxn(1− xn)Coefficient vector: (0, r,−r, 0, . . . )We can recover the system equation in chaotic regime taking about10 sample pairs or more.

0 5 10 15 20 25 30 350.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

n

xn

Sampl ing the logi sti c map, m = 10

0 5 10 15 20 25 30 3510

−8

10−6

10−4

10−2

100

102

104

106

m

||c∗−

c||2

Recovery error for logistic map, r = 3.7



5 10 15 20 25 30 35

2.4

2.5

2.6

2.7

2.8

2.9

3

3.1

3.2

3.3

3.4

3.5

3.6

3.7

3.8

3.9

4

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

Sensitive to the dynamics determined by r.

(Bifurcation diagram: Wikipedia).



5 10 15 20 25 30 35

2.4

2.5

2.6

2.7

2.8

2.9

3

3.1

3.2

3.3

3.4

3.5

3.6

3.7

3.8

3.9

4

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

r=2.5

r=3.0

r=3.2

r=3.5

3

13

3

17

Right: number of unique system values (to 2 digits).

When system is too uniform, taking more samples is just morecopies of the same data, doesn’t improve the decoding.

Chaotic regime is best in this case.


Parameter Estimation vs. System Identification

In the context of the previous example:

“Parameter Estimation”:

Assume a-priori that f(xn) = rxn − rx2n, then estimate r.

“System Identification”:

No a-priori assumption of non-zero terms.

Assume f(x) =N∑

i=0

αixi

Find non-zero αi and their values.



Autonomous 2d discrete dynamical map

[xn+1

yn+1

]=[f1(xn, yn)f2(xn, yn)

]Assume fk have form

∑i+j=0..Q

αkijx

iyj

Measure system values at ni, i = 1..mTry to recover the αk

ij .

Analogous to the setup of the 2d sparse polynomial discovery, butwe repeat the process twice: X → f1, Y → f2

X =

xi1+1

xi2+1...

xim+1

=

1 xi1 yi1 xi1yi1 . . .1 xi2 yi2 xi2yi2 . . ....1 xim yim ximyim . . .

α100

α110

α111

α120

α102...



Analogous to the setup of the 2d sparse polynomial discovery, butwe repeat the process twice: X → f1, Y → f2

X =

xi1+1

xi2+1

...

...xim+1

=

1 xi1 yi1 xi1yi1 . . .1 xi2 yi2 xi2yi2 . . .......1 xim yim ximyim . . .

α100

α110

α111

α120

α102...

Y =

yi1+1

yi2+1

...

...yim+1

=

1 xi1 yi1 xi1yi1 . . .1 xi2 yi2 xi2yi2 . . .......1 xim

yimxim

yim. . .

α200

α210

α211

α220

α202...


Example - Tinkerbell Map

xn+1 = x2n − y2

n + axn + byn

yn+1 = 2xnyn + cxn + dyn

−1.5 −1 −0.5 0 0.5−2

−1.5

−1

−0.5

0

0.5

1

x

y

Tinkerbell map in chaotic regime

Coefficient vectors:

For x: (0, a, b, 1, 0,−1, 0, . . . )For y: (0, c, d, 0, 2, 0, . . . )



0 5 10 15 20 25 30 35 4010

−10

10−5

100

105

||α∗ x−

αx||

2

Recovery of system equations, Tinkerbe l l map, a = 0.9, b = −0.6, c = 2, d = 0

0 5 10 15 20 25 30 35 4010

−10

10−5

100

105

||α∗ y−

αy||

2

m

For these system parameters, recovery works well.

With various other parameters not shown, similar results, as long assystem is in a stable regime.



Autonomous 3d dynamical systemdf

(dx,dy,dz)(x, y, z) =∑

i+j+k=0..Q

(x,y,z)αijk x

iyjzk

Now a continuous system.

Measure system values and derivatives at ti, i = 1..mEssentially the same problem setup as 3d polynomial discovery.

Perform recovery for each X → dfdx , Y → df

dy , Z → dfdz

X =

266666666664

dfdx

(x1, y1, z1)dfdx

(x2, y2, z2)

.

.

.

.

.

.dfdx

(xN , yN , zN )

377777777775=

26666641 x1 y1 z1 x1y1 x1z1 y1z1 x2

1 y21 . . .

1 x2 y2 z2 x2y2 x2z2 y2z2 x22 y22 . . .

.

.

.

.

.

.

1 xm ym zm xmym xmzm ymzm x2m y2m . . .

3777775

26666666664

α000α100α010α001α110α101

.

.

.

37777777775


Example - Rossler System

dx

dt= −y − z

dy

dt= x+ ay

dz

dt= b+ z(x− c)

Coefficient vectors:

For x: (0, 0,−1,−1, 0, 0, 0, . . . )For y: (0, 1, a, 0, 0, 0, . . . )For z: (b, 0, 0,−c, 0, 1, . . . )One difference: x, y, z not in (−1, 1), columns of A get really bad.

Normalize each column of Φ by√∑m

j=1 φ2i (tj), rescale α

accordingly after reconstruction.


Example - Rossler System

−40 −20 0 20 40−30

−20

−10

0

10

20

30

x

y

Sampling the Rossler System, m = 25

−40 −20 0 20 400

10

20

30

40

50

60

x

z



||α∗ − α||2

0 5 10 15 20 25 30 3510

−10

100

1010

dx/d

t

0 5 10 15 20 25 30 3510

−10

100

1010

dy/d

t

0 5 10 15 20 25 30 3510

−10

100

1010

dz/d

t

m

Recovery works well here in the chaotic regime:a = 0.1, b = 0.1, c = 18.Similar results in stable periodic regimes.Large number of samples required for z recovery due to the briefnessof excursions away from z ≈ 0 (see figure on previous page).


Summary of ODE System Identification

Can recover sparse dynamical system equations with a handful ofdata points.

Highly stable periodic behaviors may cause issues.

Number of non-zero terms in power series expansion needs to be lowdue to Vandermonde basis.

Can exploit dimension of system to avoid use of higher order terms.

Noise is an issue that needs to be explored.


PDE System Identification: the idea

Measure a system at m points in time and space.

Determine the system of equations governing the behavior, includingparameter values.

Inspired by Prof Platte’s prompt “pode fazer com PDE’s?”

(in reference to the ODE version of the problem: Wang, Yang, Lai,Kovanis, Grebogi)


PDE Problem Setup

Will work with 2 spatial dimensions.

Assume u(x, y, t) solves a 2nd order PDE and satisfies:

uxx = α1ut + α2ux + α3uy + α4uyy + α5uxy + α6uyt + α7utt

Want to measure u(xi, yi, ti)mi=1 and determine the αj .

Will assume instead the ability to measure the derivatives of u.

y =

266664uxx(x1, y1, t1)uxx(x2, y2, t2)

.

.

.uxx(xm, ym, tm)

377775 =

@(x1, y1, t1)@(x2, y2, t2)

.

.

.@(xm, ym, tm)

266664ut ux uy uyy uxy uyt uttut ux uy uyy uxy uyt utt

.

.

.ut ux uy uyy uxy uyt utt

377775266664α1α2

.

.

.α7

377775


Example: Heat Equation

uxx = ut − uyy, (x, y) ∈ (0, 1)× (0, 1)u(0, y, t) = u(1, y, t) = u(x, 0, t) = u(x, 1, t) = 0Identification of heat system:

0 10 20 30 40 50 60

−20

−15

−10

−5

0

5

# samples

log(||

x∗−

x||

2)

Recovery error for heat equation with 3 modes in initial conditions100 experiments per data point

@(xi,yi,ti)mi=1

ux, uxx, uy, uyy, ...

min `1

uxx = ut − uyy


Example: Wave Equation

uxx = utt − uyy, (x, y) ∈ (0, 1)× (0, 1)u(0, y, t) = u(1, y, t) = u(x, 0, t) = u(x, 1, t) = 0Identification of wave system:

0 10 20 30 40 50 60

−20

−15

−10

−5

0

5

# samples

log(||

x∗−

x||

2)

Recovery error for wave equation with 3 modes in initial conditions100 experiments per data point

@(xi,yi,ti)mi=1


min `1

uxx = utt − uyy


Example: Euler-Tricomi Equation

uxx = xuyy, (x, y) ∈ (0, 1)× (0, 1)No time in this system, so leave time derivatives out of the process.

Coefficient vector now includes terms of form xux:

uxx = α1ux + α2uy + α3uyy + α4uxy

+ x(α5ux + α6uy + α7uyy + α8uxy)+ y(α9ux + α10uy + α11uyy + α12uxy)

+ x2(α13ux + α14uy + α15uyy + α16uxy)

+ y2(α17ux + α18uy + α19uyy + α20uxy)

And adjust A accordingly.


Example: Euler-Tricomi Equation

uxx = xuyy, (x, y) ∈ (0, 1)× (0, 1)Identification of ET system:

0 10 20 30 40 50 60

−20

−15

−10

−5

0

5

# samples

log(||

x∗−

x||

2)

Recovery error for Euler−Tricomi equation100 experiments per data point

@(xi,yi,ti)mi=1


min `1

uxx = utt − uyy

Reverse Engineering Dynamical Systems From Time-Series ... › ~platte › apm598 ›...

Documents

Transcript of Reverse Engineering Dynamical Systems From Time-Series ... › ~platte › apm598 ›...