U Kang

Introduction to Data Mining

Time Series Analysis

U KangSeoul National University

U Kang

In This Lecture

Motivation of time series analysis

Similarity Search

Linear Forecasting

U Kang

Outline

Motivation

Similarity Search

Linear Forecasting

U Kang

Problem Definition

Given: one or more sequences

x1 , x2 , … , xt , …

(y1, y2, … , yt, …)

Task

Find similar sequences

Forecast future values

Classify sequences (e.g., fault or normal)

U Kang

Applications

Financial, sales, economic series

Healthcare

ECG; blood pressure monitoring

Reaction to new drugs

Elderly care

Smart house

Monitor temperature, humidity, air quality, etc.

Fault detection and prediction

Video surveillance

U Kang

Applications

Civil / automobile infrastructure

Bridge vibrations

Road conditions / traffic monitoring

Weather, environment / anti-pollution

Vibration peaks caused by the heavy loaded truck during night time in Haldiram Bridge, Kolkata

https://www.signaguard.com/bridge-health-monitoring-system/

U Kang

Outline

Motivation

Similarity Search

Linear Forecasting

U Kang

Similarity Search

Given a time series A, find the most similar time series

Applications

Clustering: group time series

Classification: classify a time series into ‘normal’ or ‘abnormal’

Rule discovery: if we observe a time series A and a time series B, it is likely to observe a time series C

Query by content: given a song A, find the most similar songs

U Kang

Importance of Distance Functions

Two major families

Euclidean and Lp norms

Time warping and variations (DTW etc.)

Euclidean: one to one Time warping: nonlinear alignments

U Kang

Euclidean and Lp norm

n

i

ii yxyxD1

2)(),(x(t) y(t)

...

pn

i

piip yxyxL /1

1

)||(),(

•L1: city-block = Manhattan

•L2 = Euclidean

•L

U Kang

Limitation of Euclidean and Lp

Cannot consider the variability in the time axis

https://www.cs.ucr.edu/~eamonn/KAIS_2004_warping.pdf

U Kang

Dynamic Time Warping

Allow accelerations - decelerations

(with or w/o penalty)

Then compute the (Euclidean) distance (+ penalty)

Related to the string-editing distance

U Kang

Dynamic Time Warping

‘stutters’:

U Kang

Applications of DTW

Bioinformatics

Medicine

Robotics

Chemistry

Gesture Recognition

…

U Kang

Dynamic Time Warping

Q: how to compute it?

A: dynamic programming

D( i, j ) = cost to match

prefix of length i of first sequence x with prefix of length j of second sequence y

U Kang

Thus, with no penalty for stutter, for sequences

x1, x2, …, xi,; y1, y2, …, yj

),1(

)1,(

)1,1(

min][][),(

jiD

jiD

jiD

jyixjiD x-stutter

y-stutter

no stutter

Dynamic Time Warping

Similar to string-edit distance

U Kang

Dynamic Time Warping

Example: DTW(x, y) 𝑥1 = 3, 𝑥2 = 5

𝑦1 = 3, 𝑦2 = 4, 𝑦3 = 5

j = 1 j = 2 j = 3

i = 1 D(i,j) = 0 D(i,j) = 1 D(i,j) = 3

i = 2 D(i,j) = 2 D(i,j) = 1 D(i,j) = 1

),1(

)1,(

)1,1(

min][][),(

jiD

jiD

jiD

jyixjiD x-stutter

y-stutter

no stutter

U Kang

Dynamic Time Warping

Complexity: O(M*N) - quadratic on the length of the strings

Many variations (penalty for stutters; limit on the number/percentage of stutters; …)

U Kang

Outline

Motivation

Similarity Search

Linear Forecasting

U Kang

Forecasting

Example: given xt-1, xt-2, …, forecast xt

0

10

20

30

40

50

60

70

80

90

1 3 5 7 9 11

Time Tick

Nu

mb

er o

f p

ack

ets

sen

t

??

U Kang

Forecasting

Solution: try to express xt

as a linear function of the past data : xt-2, xt-2, …,

(up to a window of w)

0102030405060708090

1 3 5 7 9 11Time Tick

??𝑥𝑡 ≈ 𝑎1𝑥𝑡−1 +⋯+ 𝑎𝑤𝑥𝑡−𝑤

U Kang

(Related: Back-cast)

Solution - interpolation: try to express xt

as a linear function of the past AND the future:

xt+1, xt+2, … xt+wfuture; xt-1, … xt-wpast

(up to windows of wpast and wfuture)

Exactly the same algorithm as in the forecasting

0102030405060708090

1 3 5 7 9 11Time Tick

??

U Kang

Linear Regression: idea

40

45

50

55

60

65

70

75

80

85

15 25 35 45

blood pressure

patient blood

pressure

height

1 27 43

2 43 54

3 54 72

… …

…

N 25 ??

height

Express what we don’t know (dependent variable) as a linear function of what we know (independent variables)

U Kang

Linear Auto Regression:

Time Packets

Sent (t-1)

Sensor

value (t)

1 - 41

2 43 54

3 54 62

… …

…

N 25 ??

U Kang

Linear Auto Regression:

40

45

50

55

60

65

70

75

80

85

15 25 35 45

Sensor value (t-1)S

en

so

r valu

e (

t)

Time Sensor

value (t-1)

Sensor

value (t)

1 - 41

2 41 54

3 54 62

… …

…

N 25 ??

• lag window = 1

• Dependent variable = Sensor value at t

• Independent variable = Sensor value at t-1

‘lag-plot’

U Kang

Window size > 1

Q: How to handle window size > 1 ?

A: Fit a hyperplane

xt-2

xt-1

xt

U Kang

Window size > 1

Q: How to handle window size > 1 ?

A: Fit a hyperplane

X[N w] a[w 1] = y[N 1]

Solving for a is an over-constrained problem (N > w)

U Kang

Window size > 1

X[N w] a[w 1] = y[N 1]

N

w

NwNN

w

w

y

y

y

a

a

a

XXX

XXX

XXX

2

1

2

1

21

22221

11211

,,,

,,,

,,,

independent variables

time

dependent

variable

U Kang

Window size > 1

X[N w] a[w 1] = y[N 1]

N

w

NwNN

w

w

y

y

y

a

a

a

XXX

XXX

XXX

2

1

2

1

21

22221

11211

,,,

,,,

,,,

independent variables

time

dependent

variable

U Kang

Solution of AR

Q: What is the best solution a for the problem

X[N w] a[w 1] = y[N 1] ?

A: Least square fit

𝒂 = 𝑋𝑇𝑋 −1(𝑋𝑇𝑦)

a is the vector that minimizes the RMSE between Xa and y

U Kang

Incremental Computation for AR Solution

Q: Can we estimate a incrementally?

A: Yes, with the ‘Recursive Least Squares’ (RLS) method

Main idea

The key step is to compute 𝑋𝑇𝑋 −1

We can update 𝑋𝑇𝑋 −1 efficiently by exploiting the special structure of 𝑋𝑇𝑋

Reference: Yi et al., Online data mining for co-evolving time sequences, ICDE 2000

U Kang

Recursive Least Squarew

NXN+1

At time N+1:

𝑥𝑁+1𝑇

Let 𝐺𝑁 = (𝑋𝑁𝑇𝑋𝑁)

−1 (``gain matrix’’)

𝐺𝑁+1 can be computed recursively from 𝐺𝑁

U Kang

RLS Example

Independent Variable

Dep

enden

t V

aria

ble

new point

U Kang

RLS Example

Independent Variable

Dep

enden

t V

aria

ble

new point

U Kang

Adaptability of RLS

Independent Variable

Dep

enden

t V

aria

ble

U Kang

Adaptability of RLS

Independent Variable

Dep

enden

t V

aria

ble

Trend change

(R)LS

with no forgetting

U Kang

Adaptability - ‘forgetting’

Independent Variable

Dep

enden

t V

aria

ble

Trend change

(R)LS

with no forgetting

(R)LS

with forgetting

RLS can easily handle forgetting

U Kang

What You Need to Know

Time Series Analysis

Many applications in finance, healthcare, smart house, fraud detection, etc.

Similarity Search

Given a time series, find the most similar time series

Forecasting

Autoregression (AR)

(Recurive) least square for AR

U Kang

Questions?