Matrix Analytic methods inMarkov Modelling

Continous Time Markov Models

• X: R -> X µ Z (integers)• X(t): state at time t• X: state space (discrete – countable)• R: real numbers (continuous time line)

Continous Time Markov Models

• X is piecewise constant• X is typically cadlag ("continue à droite, limite

à gauche")= RCLL (“right continuous with left limits”)

Transition probabilities

• P(X(t+h)=j|X(t)=i) ¼ h ¸ij for i j

• P(X(t+h)=j|X(t)=j) = 1- i j P(X(t+h)=i|X(t)=j) ¼ 1- hi j ¸ji

• P(X(t+h)=j)= i P(X(t+h)=j and X(h)=i)= i P(X(t+h)=j | X(h)=i) P(X(h)=i)= i j ¸ij h P(X(h)=i) + h(1- i j ¸ji) P(X(h)=j)

Transition probabilities

• P(X(t+h)=j)=i P(X(t+h)=j and X(h)=i)+ (1- k j ¸ik) P(X(h)=j)• P(t)=[P(X(t)=0) P(X(t)=1) P(X(t)=2) ..]• P(t+h) ¼ P(t)H• H=I+hQ• Qij=¸ij• Qjj= 1- i j ¸ji

Taking limits

• P(t+h) ¼ P(t)H• H=I+hQ• P(t+h) ¼ P(t)(I+hQ)=P(t)+hP(t)Q• (P(t+h)-P(t))/h ¼ P(t)Q• d/dt P(t) = P(t)Q• P(t)=P(0)exp(Qt)

Irreducibility

• X is irreducible when states are mutually reachable.

• X is irreducible iff for every i,j 2 X there is a sequence {i(1),i(2),..,I(N) 2 X} such that i(1)=i i(N)=j and ¸i(k),i(k+1) > 0 for every k 2 1,..,N-1

Recurrence• Assume X(t n -)=j and X(tn)=i j then tn is a transition time• Let {tn} be the sequence of consequetive transition times.• {Xn=X(tn)} is called the embedded chain• Let t0=0, X0=i then

k(i)=inf{k>0: X(k)=i}=inf{k>1: X(k)=i}

• Ti=tk(i)

• Ti is the time to next visit at i• X is recurrent if P(Ti<1)=1 for all i

(almost certain return to all states)• X is positive recurrent if E(Ti) · 1 for all i

Stationary probability

• d/dt P(t) = P(t)Q• P(t)=P(0)exp(Qt)• When X is irreducible and positive recurrent

there is a unique probability vector ¦ such that P(t) -> ¦

• ¦ solves ¦ Q=0

Stationary probability

• Ergodicity: ¦i = E(Di)/E(Ti)• ¦i is the fraction of the time in state i• Statistically intuitively appealing

X(t) ==i

Ti

Ditime

ExamplePoisson Counting Process

• Qi,i+1=¸

¸ ¸ ¸ ¸10 2 3

• Counts Poisson events• Birth Chain• Not irreducible• Not recurrent

ExampleBirth/Death(BD)-chain

• Qi,i+1=¸• Qi+1,i=¹

¸ ¸ ¸ ¸10 2 3

• Models a queueing system with Poisson arrival process and independent exponentially distributed service times

• ¸ is arrival rate• ¹ is service rate• Irreducible• Positive recurrent for ¸<¹

¹ ¹ ¹ ¹

ExampleBirth/Death(BD)-chain ¸ ¸ ¸ ¸10 2 3

• ½ = ¸/¹• ¦n=½¦n-1

• ¦n=½n ¦0• P0=1/n=0

1 ½n =1-½• ¦n=½n (1-½)

• E(X) = n=01 n ¦n = ½/(1-½)

¹ ¹ ¹ ¹

ExampleBirth/Death(BD)-chain ¸1 ¸2 ¸3 ¸410 2 3

• ½n = ¸n/¹n

• ¦n=½n ¦n-1

• ¦n=¦i=1n ½i ¦0

• P0=1/n=01 ¦i=1

n ½i

¹1 ¹2 ¹3 ¹4

Markov Modulated Poisson Process

• Has two modes: Modes={ON,OFF}• M(t) 2 Modes is a two state CTMC• Transmits with rate ¸ in ON mode.• Counting proces combines state spaces, i.e.

X = Modes £ {0,1,2,..}• Q(ON,i),(ON,i+1)=¸• Q(ON,i),(OFF,i)=¯• Q(OFF,i),(ON,i)=®• Qi,j=0 otherwise

Markov Modulated Poisson Process

• Q(ON,i),(ON,i+1)=¸• Q(ON,i),(OFF,i)=¯• Q(OFF,i),(ON,i)=®• Qi,j=0 otherwise

1OFF 2 3

¸ ¸ ¸ ¸10 2 3

0

ON

®¯ ® ® ®¯ ¯ ¯

MMPP with exponential service• Q(ON,i),(ON,i+1)=¸• Q(ON,i),(OFF,i)=¯• Q(OFF,i),(ON,i)=®• Q(ON,i),(ON,i-1)=¹• Q(OFF,i),(OFF,i-1)=¹• Qi,j=0 otherwise

1OFF 2 3

¸ ¸ ¸ ¸10 2 3

0

ON

®¯ ® ® ®¯ ¯ ¯

¹ ¹ ¹ ¹

¹¹¹¹

State ordering

• For a state (i,M) we denote i the level of the state

• We order states so that equal levels are gathered

• (i,OFF),(i,ON)(i+1,OFF)(i+1,ON)(i+2,OFF)(i+2,ON)

Generator matrix

)()(

)()(

)()(

)()(

)(

Q

Sub matrices

)(0

A

)()(

A

B

C

Generator matrix by submatrices

ABCAB

CABCAB

CA

Q

0

.....3210 PPPPP

),(),( ONiPOFFiPPi

Balance equations:

P0 A0 + P1 B = 0 eq(0)P0 C + P1 A + P2 B = 0 eq(1)Pi C + Pi+1 A + Pi+2 B = 0 eq(i+1)

We look for a matrix geometric solution, i.e.P1=P0 RP i+1= Pi R

Inserting in eq(0):P0 A0 + P0 R B = 0

and eq(i+1)Pi(C+R A + R2B)=0 for all Pi

Solving for R

• Pi (C+R A + R2 B)=0 for all Pi

• Sufficient that C+R A + R2 B=0 (Ricatti equation)

• Iterative solutionR0=0repeatRn+1=-(C+Rn

2 B) A-1

• Converges for irreducible positive recurrent Q

MM - service• Q(ON,i),(ON,i+1)=¸• Q(OFF,i),(OFF,i+1)=¸• Q(ON,i),(OFF,i)=¯• Q(OFF,i),(ON,i)=®• Q(ON,i),(ON,i-1)=¹• Q(OFF,i),(OFF,i-1)=0• Qi,j=0 otherwise

1OFF 2 3

¸ ¸ ¸ ¸10 2 3

0

ON

®¯ ® ® ®¯ ¯ ¯

¹ ¹ ¹ ¹

¸ ¸ ¸ ¸

Generator matrix

)()(

)()(

)()(

)()(

)()(

Q

Sub matrices

)()(

0 A

)()(

A

B

C

Generally

12344

01233

0122

011

00

AAAABAAAAB

AAABAAB

AB

Q

We still look for a matrix geometric solution:

Now: i=0 1 Ri Ai = 0 A0 + R A1 i=2 1 Ri Ai

Iteration: Rn = -A1

-1 (A0 + i=21 Rn-1

i Ai)

Solving for P0:P0 i=0

1 Ri Bi = 0

Conditions for solution:Irreducibility and pos. recurrence

Miniproject (i)• Let traffic be generated by an on/off Markov process with

on rate: ¸=1, mean rate 0.1 ¸, average on time: T=1• Let service be exponential with rate ¹ = 0.2¸• Construct the generator matrix for the data given above.• Use the iterative algoritme to solve for the load matrix R• Solve for P0 and Pi

• Compute the mean queue length• Compare with M/M/1 results

Miniproject (ii)

• Collect file size or web session duration data• Check for power tails and estimate tail power• Find appropriate parameters for a

hyperexponential approximation of the reliability estimated reliability

• Construct the generator matrix of an equivalent ME/M/1 queue